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14.  ABSTRACT 

The  utility  helicopter  (UH)-60M  Product  Manager  requested  the  U.S.  Army  Research  Laboratory’s  Human  Research  and  Engineering 
Directorate  to  participate  in  the  Limited  User  Test  for  the  UH-60M  Black  Hawk.  ARL  conducted  a  human  factors  evaluation  during  the  LUT, 
which  assessed  workload,  situation  awareness,  simulator  sickness,  pilot-vehicle  interface,  and  eye  tracker  data.  The  data  were  used  to  identify 
characteristics  of  the  UH-60M  that  enhance  or  degrade  pilot  performance.  Characteristics  that  degrade  pilot  performance  were  included  in  the 
Manpower  and  Personnel  Integration  (MANPRINT)  assessment  for  the  system’s  milestone  decision  and  should  be  considered  for  future  design 
changes  at  the  earliest  opportunity. 

Three  UH-60  crews  (six  pilots)  each  conducted  six  mission  scenarios  for  a  total  of  18  flights.  The  conditions  of  each  mission  were 
systematically  varied  and  designed  to  become  progressively  more  difficult  as  the  pilots  became  more  proficient  at  flying  the  aircraft.  The 
pilots  completed  the  simulator  sickness  questionnaire  before  and  after  each  flight.  They  completed  the  Bedford  Workload  Rating  Scale, 
Situation  Awareness  Rating  Technique,  and  the  Pilot-Vehicle  Interface  Questionnaire  after  each  mission.  In  addition  to  pilot  data,  a  tactical 
steering  committee  (TSC)  performed  an  independent  assessment  of  workload,  situation  awareness,  and  mission  success.  The  TSC  completed  a 
survey  after  each  mission.  The  data  were  analyzed  with  the  use  of  the  Wilcoxon  Signed  Ranks  Test  to  compare  pilot  ratings  between  seat 
position  and  results  between  UH-60M  and  UH-60A/L  model  aircraft. 

The  mean  workload  rating  for  all  tasks  for  the  UH-60M  was  2.71,  indicating  that  the  pilots  typically  had  enough  workload  capacity  for  all 
desirable  additional  tasks.  The  mean  situation  awareness  rating  provided  by  the  pilots  was  28.25.  This  SA  rating  indicates  that  the  pilots  felt 
they  had  high  levels  of  situation  awareness  during  the  missions.  The  pilots  also  provided  data  and  comments  regarding  the  pilot-vehicle 
interface  and  offered  recommendations  for  design  improvements.  Finally,  the  eye  tracker  results  showed  that  the  flying  pilot  was  focused  out 
the  window  85.60%  of  the  time  while  the  non-flying  pilot  spent  only  28.21%  focused  out  the  window. 

The  results  indicated  that  the  UH-60M  crew  station  resulted  in  acceptable  workload  and  SA  levels  and  offers  significant  improvements 
compared  to  the  UH-60A/L.  However,  several  issues  were  identified  which,  if  corrected,  could  offer  further  reductions  in  workload  and 
improve  pilot  performance.  These  issues  should  be  considered  for  future  modifications  of  the  UH-60M  design,  and  future  human  factors 
evaluations  should  be  conducted  to  evaluate  the  effectiveness  of  any  design  changes. 
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1.  Introduction 


1.1  Background  and  Purpose 

The  utility  helicopter  (UH)-60  Black  Hawk  is  a  twin-turbine  engine,  single  rotor,  semimonocoque 
fuselage,  rotary  wing  helicopter  capable  of  transporting  cargo,  1 1  combat  troops,  and  weapons 
during  day,  night,  instrument  meteorological  conditions  (IMC),  visual  meteorological  conditions 
(VMC),  and  degraded  visual  environmental  conditions  (see  figure  1).  The  main  and  tail  rotor 
systems  consist  of  four  blades  each,  with  the  capability  to  manually  fold  the  main  rotor  blades, 
scissor  the  tail  rotor  paddles,  and  fold  the  tail  pylon  assembly  for  deployment,  transport,  or 
storage.  A  movable,  horizontal  folding  stabilator  assembly  is  situated  on  the  lower  portion  of  the 
tail  rotor  pylon  to  provide  enhanced  flight  characteristics. 


Figure  1.  UH-60M  Black  Flawk  helicopter. 


The  UH-60  Black  Hawk  helicopter  provides  air  assault,  general  support,  and  medical  evacuation 
(MEDEVAC)  capabilities  for  the  U.S.  Army.  The  UH-60  also  supports  the  Army  Airborne 
Command  and  Control  System  and  special  operations.  The  UH-60A  and  UH-60L  model  Black 
Hawk  helicopters  were  first  fielded  in  the  1970s  and  are  approaching  the  end  of  their  useful 
service  life.  Increasing  operations  and  support  costs  and  decreasing  operational  readiness  are 
consequences  of  the  aging  fleet.  The  UH-60M  program,  formerly  a  recapitalization  program  of 
existing  airframes,  is  now  a  new  production  program  designed  to  improve  the  life  of  the  current 
system,  reduce  operations  and  support  costs,  and  increase  operational  readiness.  Additionally, 
the  UH-60M  will  meet  future  digitization  and  situational  awareness  (SA)  requirements,  increase 
the  lift  and  range  capabilities  of  the  current  aircraft,  and  provide  an  improved  platform  for  the 
HH-60M  MEDEVAC  helicopter. 

Many  significant  changes  in  the  helicopter  are  in  the  UH-60M  cockpit  (see  figure  2).  Four 
multi-function  displays  (MFDs)  are  placed  in  a  smaller  dashboard,  6  inches  narrower  than  on  the 
current  UH-60,  which  provides  additional  visibility  outside  the  cockpit,  thereby  enhancing 
safety.  The  MFDs  will  display  primary  flight  instruments  that  replace  the  traditional  analog 
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instruments  and  a  digital  map  that  will  provide  tactical  and  navigational  information  to  the  pilots, 
significantly  enhancing  SA.  In  addition,  a  flight  management  system  (FMS)  is  used  to  manage 
voice  and  digital  communication,  navigation,  and  flight  planning. 


Figure  2.  Artist’s  rendering  of  the  UH-60M  Black  Hawk  crew  station. 


The  U.S.  Army  Operational  Test  Command  conducted  a  successful  Limited  User  Test  (LUT)  in 
support  of  the  low  rate  initial  production  decision  in  the  second  quarter  of  fiscal  year  2005.  The 
LUT  was  focused  on  evaluating  the  pilot-vehicle  interface  (PVI),  and  the  primary  purpose  was  to 
provide  input  to  the  U.S.  Army  Evaluation  Center’s  system  evaluation  report.  The  data  were 
also  used  by  ARL’s  Human  Research  and  Engineering  Directorate  (HRED)  to  draft  the 
Manpower  and  Personnel  Integration  (MANPRINT)  assessment. 

At  the  request  of  the  UH-60M  Project  Manager’s  Office  (PMO),  ARL’s  HRED  conducted  a 
human  factors  evaluation  (HFE)  of  the  UH-60M  crew  station  during  the  LUT.  The  HFE  focused 
on  workload,  SA,  and  PVI.  Additional  data  collection  included  eye  tracker  data,  simulator 
sickness  data,  and  tactical  steering  committee  (TSC)  ratings  of  workload,  SA,  and  mission 
success. 

The  purpose  of  this  report  is  to  summarize  the  human  factors  data  collected  during  the  LUT  by 
ARL. 
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1.2  Assessment  of  Crew  Workload 

A  common  definition  of  pilot  workload  is  “the  integrated  mental  and  physical  effort  required  to 
satisfy  the  perceived  demands  of  a  specified  flight  task”  (Roscoe,  1985).  It  is  important  to  assess 
pilot  workload  because  mission  accomplishment  is  related  to  the  mental  and  physical  ability  of 
the  crew  to  effectively  perform  their  flight  and  mission  tasks.  If  one  or  both  pilots  experience 
excessively  high  workload  while  performing  flight  and  mission  tasks,  the  tasks  may  be 
performed  ineffectively  or  abandoned.  In  order  to  assess  whether  the  pilots  are  task  overloaded 
during  the  mission  profiles,  the  level  of  workload  for  each  pilot  must  be  evaluated. 

1.2.1  Bedford  Workload  Rating  Scale 

The  pilots  completed  the  Bedford  Workload  Rating  Scale  (BWRS)  (appendix  A)  immediately 
after  each  mission  to  estimate  the  level  of  workload  that  they  experienced  during  missions.  The 
pilots  also  provided  workload  estimates  for  what  they  felt  the  workload  would  be  performing  the 
same  tasks  in  a  UH-60A/L.  The  pilots  used  the  BWRS  to  rate  the  workload  needed  to 
accomplish  32  UH-60M  aircrew  training  manual  (ATM)  tasks  (appendix  A).  The  ATM  tasks 
were  selected  by  personnel  from  ARL,  the  UH-60M  PMO,  and  the  Training  and  Doctrine 
Command  (TRADOC)  System  Manager  (TSM)  for  utility  aircraft  because  they  were  estimated 
to  have  the  most  impact  on  pilot  workload  during  the  planned  missions. 

The  BWRS  has  been  used  extensively  by  the  military,  civil,  and  commercial  aviation 
communities  for  pilot  workload  estimation  (Roscoe  &  Ellis,  1990).  It  requires  pilots  to  rate  the 
level  of  workload  associated  with  a  task,  based  on  the  amount  of  spare  capacity  they  feel  they 
have  to  perfonn  additional  tasks.  Spare  workload  capacity  is  an  important  commodity  for  pilots 
because  they  are  often  required  to  perform  several  tasks  concurrently.  For  example,  pilots  often 
perform  navigational  tasks,  communicate  via  multiple  radios,  monitor  aircraft  systems,  and  assist 
the  pilot  with  the  controls  with  flight  tasks  (e.g.,  maintain  air  space  surveillance)  within  the  same 
time  interval.  Mission  performance  is  reduced  if  pilots  are  task  saturated  and  have  little  or  no 
spare  capacity  to  perfonn  other  tasks.  Integration  of  the  UH-60M  crew  station  should  help 
ensure  that  pilots  can  maintain  adequate  spare  workload  capacity  while  performing  flight  and 
mission  tasks. 

1.3  Assessment  of  Crew  Situational  Awareness  (SA) 

SA  can  be  defined  as  the  pilot’s  mental  mode  of  the  current  state  of  the  flight  and  mission 
environment.  A  more  formal  definition  is  “the  perception  of  the  elements  in  the  environment 
within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of 
their  status  in  the  near  future”  (Endsley,  1988).  It  was  important  to  assess  SA  during  the  LUT 
because  it  had  a  direct  impact  on  pilot  and  system  performance.  Good  SA  should  increase  the 
probability  of  good  decision  making  and  performance  by  air  crews  when  they  perform  flight  and 
mission  tasks  in  the  UH-60M. 
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1.3.1  Situation  Awareness  Rating  Technique  (SART) 

The  SART  (appendix  B)  is  a  multi-dimensional  rating  scale  for  operators  to  report  their 
perceived  SA.  The  SART  was  developed  as  an  evaluation  tool  for  the  design  of  air  crew  systems 
(Taylor,  1989)  and  examines  three  components  of  SA:  understanding,  supply,  and  demand. 
Taylor  proposed  that  SA  depends  on  the  pilot’s  understanding  (U)  (e.g.,  quality  of  information 
he  receives),  and  the  difference  between  the  demand  (D)  on  the  pilot’s  resources  (e.g., 
complexity  of  mission)  and  the  pilot’s  supply  (S)  (e.g.,  ability  to  concentrate).  When  D  exceeds 
S,  there  is  a  negative  effect  on  U  and  an  overall  reduction  of  SA.  The  formula  SA  =  U  -  (D  -  S) 
is  used  to  derive  the  overall  SART  score.  The  SART  is  one  of  the  most  thoroughly  tested  rating 
scales  for  estimating  SA  (Endsley,  2000). 

1.4  Assessment  of  Pilot-Vehicle  Interface  (PVI) 

The  crew  station  PVI  directly  impacts  crew  workload  and  SA  during  a  mission.  A  crew  station 
that  is  designed  to  augment  the  cognitive  and  physical  abilities  of  crews  will  minimize  workload, 
enhance  SA,  and  contribute  to  successful  mission  performance.  The  pilots  completed  a  PVI 
questionnaire  after  each  mission  (appendix  C)  to  identify  any  problems  with  the  usability  of  the 
controls,  displays,  or  subsystems. 

1.5  Assessment  of  Simulator  Sickness 

Simulator  sickness  has  been  defined  as  a  condition  when  pilots  suffer  physiological  discomfort 
in  the  simulator  but  not  while  flying  the  actual  aircraft  (Kennedy,  Lilienthal,  Berbaum,  Balzley, 

&  McCauley,  1989).  It  is  generally  believed  that  simulator  sickness  is  caused  by  a  mismatch 
between  the  visual  and  vestibular  sources  of  information  about  self-motion  or  between  the 
sensory  information  (e.g.,  acceleration  cues)  presented  by  the  simulator  and  the  sensory 
information  presented  by  the  primary  aircraft  that  the  pilot  operates.  When  the  sensory 
information  presented  by  the  simulator  does  not  match  the  aircraft,  the  pilot’s  nervous  system 
reacts  adversely  to  the  sensory  mismatch  and  the  pilot  begins  to  experience  discomfort. 
Characteristics  of  simulator  sickness  include  nausea,  dizziness,  drowsiness,  and  several  other 
symptoms  (Kennedy  et  ah,  1989).  It  is  important  to  assess  simulator  sickness  because  the 
discomfort  felt  by  pilots  can  be  distracting.  Pilot  distraction  is  one  of  the  operational 
consequences  of  simulator  sickness  listed  by  Crowley  (1987).  If  pilots  are  distracted  by  the 
discomfort  they  feel  during  missions,  their  perfonnance  is  likely  to  suffer.  Additionally,  the 
discomfort  could  influence  the  perceived  levels  of  workload  and  SA  that  the  pilots  experienced 
during  a  mission. 

1.5.1  Simulator  Sickness  Questionnaire  (SSQ) 

The  SSQ  (appendix  D)  was  administered  to  the  pilots  to  estimate  the  severity  of  physiological 
discomfort  that  they  experienced  during  missions  and  to  help  assess  whether  they  were  being 
distracted  by  the  discomfort.  The  SSQ  (Kennedy,  Lane,  Berbaum,  &  Lilienthal,  1993)  is  a 
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checklist  of  16  symptoms  that  are  categorized  into  three  subscales:  oculomotor  (e.g.,  eyestrain, 
difficulty  focusing,  blurred  vision),  disorientation  (e.g.,  dizziness,  vertigo),  and  nausea  (e.g., 
nausea,  increased  salivation,  burping).  The  three  subscales  are  combined  to  produce  a  total 
severity  score.  This  score  is  an  indicator  of  the  overall  discomfort  that  the  pilots  experienced 
during  the  mission. 

1.6  Tactical  Steering  Committee  (TSC) 

A  TSC  observed  each  mission  and  rated  crew  workload,  crew  SA,  crew  coordination,  and 
mission  success  (appendix  E).  The  TSC  provided  an  independent  assessment  of  the  workload 
and  SA  levels  experienced  by  the  crews.  They  also  helped  identify  whether  problems  with  crew 
workload  or  crew  SA  contributed  to  lack  of  mission  success. 

TSC  personnel  observed  each  mission  from  the  battle  master  station  in  the  Systems  Integration 
Laboratory  (SIL)  located  at  the  Software  Engineering  Directorate  (SED)  of  the  U.S.  Army 
Aviation  and  Missile  Command  (AMCOM)  at  Redstone  Arsenal,  Alabama,  where  they  could 
observe  crew  station  displays  and  the  out-the-window  (OTW)  view  provided  to  the  crew  (see 
figure  3).  They  also  listened  to  all  audio  communications  between  crew  members  and  outside 
sources  during  the  missions.  A  large  projection  map  provided  real-time  status  of  the  location  of 
the  aircraft  on  the  terrain  database. 


Figure  3.  Battle  master  station. 


1.7  Aviation  and  Missile  Command,  Software  Engineering  Directorate 

The  SIL  offered  the  appropriate  capability  required  to  conduct  the  LUT  in  a  simulated 
environment.  SED  is  a  recognized  leader  in  supporting  the  acquisition,  research,  development, 
and  sustainment  of  some  of  many  sophisticated  weapon  systems.  SED  maintains  expertise  in  the 
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Army’s  prevailing  policies  on  acquisition.  These  policies  include  software  reuse,  software 
metrics,  post-deployment  software  support,  process  improvement,  computer  resource  margins 
analysis,  and  risk  management. 

SED’s  risk-based  approach  to  perfonning  verification  and  validation  (V&V)  is  designed  to  focus 
on  identified  problem  areas,  to  ensure  effective  software  engineering  support  with  minimum 
cost.  Using  the  facilities  and  the  numerous  tactical  hardware  and  software  laboratories,  SED 
provides  weapon  systems  with  the  highest  quality  support  in  the  areas  of  joint  interoperability 
testing  and  engineering. 


1.7.1  Systems  Integration  Laboratory  (SIL) 


One  of  the  simulation  resources  within  the  SED  is  the  SIL.  The  UH-60M  SIL  was  composed  of 
four  main  components:  tactical  systems,  data  monitoring  and  collection  systems,  user  control 
system,  and  simulation  systems  (see  figure  4).  When  combined,  these  components  provided  a 
UH-60M  cockpit  environment  coupled  to  the  aircraft’s  external  environment.  The  cockpit  was 
incorporated  into  the  forward  section  of  a  UH-60L  aircraft.  Using  a  section  of  the  actual  aircraft 
enabled  us  to  provide  a  realistic  environment  with  production-representative  hardware.  This 
enabled  the  human  factors  and  MANPRINT  experts  to  provide  a  thorough  evaluation  during  the 
LUT  (see  figure  5). 


Image  Generation  j 
System 


Virtual  Battlefield  (DIS) 


Jbl 

□  J 

TT‘  f* 
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CTB  User  Control  System 


UH-60M  CTB  Architecture 

Display  System 


Tactical  Radios 


6-DoF  Flight  Simulation 


Tactical  Component  Data  Monitoring  & 

Simulation  Suite  Collection  System 

(component  sims)  (video/audio,  JVMF,  OTB) 


Figure  4.  UH-60M  SIL  architecture. 
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Figure  5.  Photograph  of  SIL  crew  station. 


2.  Method 


2.1  Participants 

Participants  consisted  of  six  male  UH-60  pilots.  One  pilot  was  a  Chief  Warrant  (CW)  3  battalion 
standards  pilot  with  the  1-212  Aviation  Regiment  at  Fort  Rucker,  Alabama.  Two  pilots  were 
instructor  pilots  from  the  same  unit  and  held  the  ranks  of  CW2  and  CW3.  One  pilot  was  a 
civilian,  instructor  pilot  for  the  maintenance  test  pilot  course  at  Fort  Rucker.  Two  pilots  were 
Army  pilots  stationed  at  Fort  Campbell,  Kentucky.  One  held  the  rank  of  captain  and  served  as 
the  assistant  S-3  officer  for  the  4-101  Aviation  Regiment.  The  other  pilot  was  a  CW2  and  was  a 
member  of  the  5-101  Aviation  Regiment.  These  pilots  represented  a  broad  range  of  experience 
with  total  flight  hours  that  ranged  from  180  to  4141.  The  demographic  characteristics  of  the 
pilots  are  listed  in  table  1 . 
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Table  1.  Pilot  demographics  (N  =  6). 


Summary  of  demographic 
characteristics 

Age 

(yrs) 

Flight  hours  in 
UH-60A/L  Black  Hawk 

Total  flight  hours 
in  Army  aircraft 

Mean 

34.3 

1760 

2059 

Median 

33.5 

1569 

2196 

Range 

25  to  46 

140  to  3848 

180  to  4141 

Anthropometric  data  were  also  collected  for  each  pilot.  Ten  critical  measurements  were  taken 
for  each  pilot,  including  stature,  bideltoid  breadth,  chest  depth,  butt-knee  length,  interpupillary 
breadth,  functional  leg  length,  hand  length,  hand  breadth,  thumb  tip  reach,  and  sitting  eye  height 
(Gordon  &  Donelson,  1991).  These  measurements  were  taken  to  ensure  that  the  participants 
represented  a  broad  range  of  the  intended  user  population  with  respect  to  human  dimensions. 
Table  2  shows  the  dimensions  for  each  anthropometric  measurement. 

The  TSC  consisted  of  two  personnel  with  extensive  Army  aviation  experience.  One  was  the 
TRADOC  System  Manager  user  representative  from  Fort  Rucker  and  held  the  rank  of  CW3. 

The  other  was  a  captain  who  was  currently  serving  as  the  Operational  Test  Command  test  officer 
for  the  UH-60M  LUT. 


Table  2.  Anthropometric  measurements. 


Pilot  1 
(cm) 

Pilot  2 
(cm) 

Pilot  3 
(cm) 

Pilot  4 
(cm) 

Pilot  5 
(cm) 

Pilot  6 
(cm) 

Percentile 

Range 

Sitting  Eye 
Height 

81.3 

81.4 

80.5 

80.5 

84.2 

82.4 

43  to  84 

Thumb  Tip 

Reach 

79.8 

86.8 

80.0 

84.2 

87.4 

83.0 

43  to  97 

Hand  Breadth 

9.0 

9.1 

8.7 

9.1 

9.6 

OO 

oo 

22  to  93 

Hand 

Length 

20.4 

23.7 

19.8 

21.4 

20.4 

18.3 

9  to  99 

Functional  Leg 
Length 

110.6 

119.7 

110.0 

110.8 

118.8 

105.1 

24  to  99 

Interpupillary 

Breadth 

61.5 

62.5 

59.0 

69.0 

65.0 

58.0 

4  to  93 

Butt-Knee 

Length 

62.5 

66.5 

62.9 

62.1 

67.4 

58.8 

12  to  98 

Chest 

Depth 

26.4 

32.0 

24.4 

28.7 

34.4 

21.8 

5  to  99 

Bideltoid 

Breadth 

49.9 

55.5 

50.1 

56.2 

51.3 

46.7 

10  to  99 

Stature 

178.4 

184.5 

179.8 

176.0 

186.4 

178.4 

44  to  93 

2.2  Data  Collection 

The  BWRS,  SART,  PVI,  SSQ,  and  TSC  questionnaires  (appendices  A  through  E)  were 
developed  in  accordance  with  published  guidelines  for  proper  fonnat  and  content  (O’Brien  & 
Charlton,  1996).  A  pre-test  was  conducted  to  refine  the  questionnaires  and  to  ensure  that  they 
could  be  easily  understood  and  completed  by  pilots  and  TSC  members. 
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The  pilots  completed  the  PVI,  workload,  and  SA  questionnaires  immediately  after  each  mission. 
The  pilots  completed  the  SSQ  before  and  after  each  mission.  TSC  members  completed  the  TSC 
questionnaire  after  each  mission.  Additional  data  were  obtained  from  the  pilots  and  TSC 
members  during  post-mission  discussions  and  the  final  after-action  review  (AAR). 

Questionnaire  results  were  clarified  with  information  obtained  during  post-mission  discussions 
and  the  daily  AARs. 

2.2.1  Eye  Tracker  System 

Although  the  data  from  the  questionnaires  were  systematically  gathered  by  widely  accepted  HFE 
methods,  they  were  still  subjective  in  nature.  Complementary  objective  data  were  collected 
through  a  head  and  eye  tracking  system  from  Applied  Science  Laboratories  (ASL).  Their  system 
was  used  because  it  was  capable  of  integrating  a  laser  head  tracker  to  allow  unrestricted  head 
movement  during  data  collection  and  it  was  compatible  with  the  head  gear  unit  (HGU)-56  flight 
helmet.  The  EyeHead1  Package  integrated  a  Model  501  eye  tracker  and  a  Laserbird2  head 
tracker.  This  technology  allowed  us  to  collect  digital  data  that  specify  point  of  gaze  with  respect 
to  stationary  objects  within  the  crew  station.  The  ASL  software  allowed  data  collectors  to 
continuously  monitor  the  eye  position  of  the  pilots  by  crosshairs  superimposed  over  live  imagery 
(see  figure  6).  The  software  also  included  a  built-in  analytical  tool  that  allowed  data  to  be 
viewed  in  tabular  or  graphical  format. 


Figure  6.  Eye  tracker  scene  camera  monitors  and  control  panel  interface. 


'EyeHead  is  a  trademark  of  ASL. 

“Laserbird  is  a  trademark  of  Ascension  Technology  Corp. 
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2.3  Data  Analysis 

Pilot  responses  to  the  BWRS,  SART,  SSQ  and  PVI  questionnaires  were  analyzed  with  means 
and  percentages.  Their  responses  to  the  BWRS,  SART,  and  SSQ  were  further  analyzed  with  the 
Wilcoxon  Signed  Ranks  Test  (WSRT)  to  compare  pilot  ratings  between  seating  position  (left 
versus  right)  and  aircraft  model  (M  versus  A/L).  The  WSRT  was  used  to  calculate  probability 
values  for  data  comparisons. 

We  summarized  the  eye  tracker  data  by  calculating  the  total  percentage  of  mission  time  spent 
focused  on  different  areas  of  interest  (AOIs).  There  were  periods  of  the  mission  when  no  eye 
tracker  data  were  collected  or  the  data  collected  were  unusable  because  the  calibration  was 
disturbed;  therefore,  for  the  purpose  of  analyzing  eye  tracker  data,  the  mission  time  is  defined  as 
the  time  during  the  mission  when  useful  eye  tracker  data  were  collected.  Four  AOIs  were 
created  for  each  pilot:  left  MFD,  right  MFD,  OTW,  and  other  (see  figure  7).  The  “other” 
category  captured  eye  fixations  focused  on  areas  not  captured  by  the  other  AOIs.  The  FMS  and 
pilot’s  kneeboard  were  both  captured  in  this  category. 


Figure  7.  Eye  tracker  areas  of  interest. 

The  eye  tracker  data  were  complicated  during  the  LUT  because  the  pilots  were  allowed  to 
alternate  flying  responsibilities  throughout  the  flight.  Since  the  eye  tracker  data  are  typically 
separated  by  flying  versus  non-flying  pilot,  this  made  analysis  of  the  data  much  more  difficult. 
Additionally,  the  mission  scenarios  included  single-ship  and  multi-ship  missions.  During  multi¬ 
ship  missions,  the  lead  aircraft  is  responsible  for  maintaining  altitude  and  air  speed.  Trail  aircraft 


10 


maintain  their  position  relative  to  the  aircraft  in  front  of  them.  We  assumed  there  would  be  a 
significant  difference  between  single-ship  and  multi-ship  eye  tracker  data,  so  the  data  were 
segregated  according  to  these  two  mission  types. 

2.4  Limitations  of  Assessment 

The  SIL  had  several  limitations  that  restricted  the  human  factors  experts  from  performing  a  full 
assessment  of  the  UH-60M  crew  station.  The  SIL  was  a  non-motion  simulator.  During  actual 
flight,  gravitational  forces  contribute  to  the  pilot’s  SA.  The  effects  of  the  motion  and  vibration 
can  also  affect  pilot  perfonnance  and  the  ability  to  clearly  read  the  aircraft  instruments.  The  lack 
of  motion  limited  the  pilot’s  ability  to  perform  a  comprehensive  evaluation  of  SA,  workload,  and 
readability  of  displays. 

The  SIL  was  not  equipped  with  a  landing  light  or  searchlight.  This  limitation  prevented  flight 
scenarios  from  being  conducted  in  night,  unaided  conditions.  The  effects  of  natural  sunlight 
could  not  be  replicated  in  the  SIL.  This  limited  the  ability  to  evaluate  the  readability  of  the 
displays  during  high  illumination  conditions.  In  addition  to  sunlight,  other  natural  environmental 
conditions,  such  as  noise  levels,  temperature,  and  humidity,  were  not  representative  of  an  actual 
aircraft  in  a  southwest  Asia  scenario. 

Only  two  types  of  joint  variable  message  format  (JVMF)  messages  were  available  during  the 
LUT.  This  limited  the  ability  to  fully  evaluate  the  workload  associated  with  sending  and 
receiving  digital  messages. 

Lack  of  available  time  and  resources  made  it  infeasible  to  conduct  simulated  flights  in  an  A  or  L 
model  UH-60  to  collect  workload  data.  As  a  result,  in  order  to  complete  a  workload  comparison 
between  the  UH-60M  and  previous  model  UH-60s,  participants  estimated  what  they  thought 
workload  might  be  for  each  task  if  they  perfonned  it  in  a  UH-60A/L.  This  lack  of  “real” 
workload  data  for  the  UH-60A/L  somewhat  limits  the  usefulness  of  the  comparison. 

2.5  Test  Schedule  and  Description  of  Mission  Scenarios 

The  first  phase  of  the  evaluation  was  pilot  training.  This  training  occurred  from  July  19  to  July 
30,  2004.  Members  of  the  PMO  conducted  the  training  to  teach  the  pilots  how  to  operate  the 
UH-60M  and  the  limitations  of  the  SIL.  Training  consisted  of  classroom  training  and  flight  time 
in  the  simulator.  The  second  phase  of  the  LUT  was  the  test  phase  that  occurred  from  August  2  to 
August  27,  2004.  Each  crew  conducted  six  vignettes  in  accordance  with  the  operational  mode 
summary  and  mission  profile  (OMS/MP).  The  six  vignettes  were  designed  to  become 
progressively  more  difficult  as  the  pilots  became  more  proficient  at  flying  the  UH-60M.  A 
description  of  the  vignettes  is  shown  in  table  3. 
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Table  3.  Vignette  descriptions  and  conditions. 


Vignette 

Number 

Description 

Conditions 

1 

Entry  into  Theater 

Day,  Single-Ship,  MOPP  0,  VMC/IMC 

2 

Air  Assault 

Day,  Multi-Ship,  MOPP  0,  VMC 

3 

Air  Movement 

Day,  Single-Ship,  MOPP  4,  VMC 

4 

Landing  Strip  Seizure 

Day,  Multi-Ship,  MOPP  4,  VMC 

5 

Air  Movement  -  High  Gross  Weight 

Night,  Single-Ship,  MOPP  0,  VMC/IMC,  NVG 

6 

Long  Range  Surveillance  Detachment 

Night,  Multi-Ship,  MOPP  4,  VMC,  NVG 

MOPP  =  mission-oriented  protective  posture 


The  test  schedule  consisted  of  one  test  flight  each  day.  The  crew  began  each  day  attending  a 
mission  briefing  presented  by  the  Directorate  of  Combat  Developments  (DCD)  user 
representative.  After  the  crew  acknowledged  the  briefing,  they  prepared  for  and  conducted  the 
mission  in  the  SIL.  Immediately  after  each  mission,  the  crew  members  were  relocated  to  a  data 
collection  room  where  they  completed  the  human  factors  surveys.  After  the  crew  completed  the 
surveys,  the  test  team  conducted  an  AAR.  At  the  end  of  the  test,  all  the  pilots  participated  in  a 
final,  comprehensive  AAR. 


3.  Results 


3.1  Crew  Workload 

3.1.1  Mean  Workload  Ratings  for  ATM  Tasks 

The  mean  overall  workload  rating  for  all  ATM  tasks  for  the  UH-60M  was  2.7 1 .  The  mean 
workload  rating  for  the  same  tasks,  assuming  they  were  performed  in  a  UH-60A/L,  was  3.99. 
This  difference  between  workload  ratings  given  for  the  UH-60M  and  the  UH-60A/L  was 
statistically  significant  (WSRT,  z  =  -2.201  ,p  =  0.028)  (see  figure  8).  The  practical  significance 
of  this  difference  is  also  noteworthy  because  the  mean  workload  rating  for  the  UH-60M  indicates 
that  workload  was  satisfactory  without  a  reduction  in  spare  workload  capacity  while  the  rating 
for  the  UH-60A/L  indicates  that  workload  is  not  satisfactory  and  results  in  a  reduction  in  spare 
workload  capacity. 

The  UH-60M  workload  ratings  were  also  compared  between  seat  positions.  The  mean  UH-60M 
workload  ratings  for  the  left  and  right  seat  positions  were  2.80  and  2.58,  respectively.  The 
difference  between  seat  positions  was  not  statistically  significant  (WSRT,  z  =  -0.943 ,p  =  0.345). 

Two  tasks  received  peak  workload  ratings  of  6,  indicating  that  workload  was  not  tolerable  for  the 
task.  These  tasks  included  maintaining  air  space  surveillance  and  transmitting  tactical  reports. 
Pilots  indicated  that  both  tasks  received  high  workload  ratings  because  of  the  difficulty  involved 
with  perfonning  JVMF  messaging,  causing  increased  workload  and  decreasing  the  time  available 
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to  perform  airspace  surveillance.  Appendix  F  includes  a  table  of  mean  workload  ratings  for  all 
tasks.  A  set  of  pilot  comments  regarding  workload  is  included  in  appendix  G. 


Figure  8.  Comparison  of  workload  ratings  for  UH-60M  versus  UH-60A/L. 


3.1.2  TSC  Ratings  for  Workload  and  Crew  Coordination 

The  TSC  provided  an  overall  mean  workload  rating  of  2.91  for  pilots  sitting  in  either  crew 
position.  An  overall  mean  rating  of  2.91  indicates  that  the  pilots  typically  had  “enough  workload 
capacity  for  all  desirable  additional  tasks”.  The  TSC  also  gave  individual  ratings  for  pilots  in 
each  seat.  The  workload  ratings  for  the  left  and  right  seats  were  2.99  and  2.83,  respectively. 
Overall,  the  results  indicate  that  workload  was  adequately  divided  between  the  two  crew 
members. 

The  TSC  also  rated  crew  coordination  for  each  mission  using  a  5-point  rating  scale.  The  mean 
crew  coordination  rating  for  all  missions  was  1.83  (see  figure  9). 


Mean  Crew 
Coordination 
Rating  (1.83) 

▼ 

1  2  3  4  5 

Excellent  Good  Average  Needs  Improvement  Unacceptable 

Figure  9.  Overall  TSC  crew  coordination  ratings. 
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3.2  Crew  Situational  Awareness 


3.2.1  SA  Ratings  by  the  Subjects 

The  overall  SART  score  provided  by  the  pilots  was  28.25  for  the  UH-60M.  This  score  indicates 
that  the  pilots  felt  they  had  high  levels  of  overall  SA  during  the  missions.  The  pilots  also 
provided  an  SA  rating  for  the  UH-60A/L  of  1 1 .69.  An  SA  rating  of  1 1 .69  indicates  that  the 
pilots  felt  they  experienced  moderate  levels  of  SA  during  the  mission.  The  difference  between 
SA  ratings  for  the  UH-60M  and  UH-60A/L  was  statistically  significant  (WSRT,  z  =  -2.201  ,p  = 
0.028)  and  is  depicted  in  figure  10.  The  pilots  indicated  that  the  improved  SA  in  the  UH-60M 
was  primarily  attributable  to  the  digital  map. 

The  SA  ratings  for  the  UH-60M  were  compared  by  seat  position.  The  SA  ratings  for  the  left  and 
right  seats  were  28.28  and  28.22,  respectively.  The  difference  between  SA  scores  for  the  left  and 
right  seats  was  not  statistically  significant  (WSRT,  z  =  -0.674,/?  =  0.500)  and  indicates  that  both 
pilots  had  high  SA  during  the  missions. 


Figure  1 1  shows  the  subscale  ratings  for  the  UFI-60M  and  UH-60A/L.  Figure  12  shows  the 
subscale  ratings  for  the  left  and  right  seats  of  the  UFI-60M.  Pilot  SA  comments  are  presented  in 
appendix  H. 
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'Demand'  During  Missions 
Low  1 - 3 - 5 - 7 - 9 - 11 - 13 - 15 - 17 - 19 - 21  High 

t 

|  UH-60A/L:  13.83  | 

'Supply'  During  Missions 
Low  1 - 4 - 7 - 10 - 13 - 16 - 19 - 21 - 24 - 28  High 

t 

UH-60A/L:  14.99  | 

'Understanding'  During  Missions 
Low  1 - 3 - 5 - 7 - 9 - 11 - 13 - 15 - 17 - 19 - 21  High 

t 

UH-60A/L:  10.53 


Figure  11.  Comparison  of  subscale  ratings  for  UH-60M  versus  UH-60A/L. 


Figure  12.  Comparison  of  subscale  ratings  for  left  versus  right  seat  (UH-60M  only). 


3.2.2  TSC  SA  Ratings 

The  TSC  provided  an  independent  assessment  of  SA  based  on  the  scale  shown  in  table  4.  The 
mean  TSC  SA  rating  was  1.47.  This  indicates  that  the  TSC  perceived  that  the  crews  typically 
had  adequate  levels  of  SA  with  some  periods  of  minor  variation  between  perception  and  reality. 
A  complete  set  of  TSC  comments  regarding  SA  is  given  in  appendix  I. 
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Table  4.  TSC  SA  ratings. 


1 

Crew  was  consistently  aware  of  all  entities  on  the  battlefield.  -4 - 

2 

Crew  was  aware  of  the  battlefield  with  minor  or  insignificant  variation  between 
perception  and  reality. 

3 

Crew  was  aware  of  the  battlefield.  Variation  between  reality  and  perception  did  not 
significantly  impact  mission  success. 

4 

SA  needs  improvement.  Lack  of  SA  had  some  negative  effect  on  the  success  of  the 
mission. 

5 

Lack  of  SA  caused  mission  failure. 

Mean  Rating 
1.47 

(SD  =  0.45) 


3.3  PVI 

The  pilots  completed  a  comprehensive  PVI  survey  after  each  mission.  This  survey  allowed  the 
pilots  to  assign  ratings  for  each  question  and  provide  comments  about  why  they  rated  the 
question  a  certain  way.  This  section  of  the  report  highlights  the  most  common  issues  that  were 
addressed  by  the  pilots.  A  complete  set  of  PVI  results  and  pilot  comments  is  included  in 
appendix  J. 

3.3.1  Joint  Variable  Message  Format  (JVMF) 

JVMF  is  a  standard  messaging  fonnat  used  in  the  UH-60M  Black  Hawk  to  exchange  digital 
messages  between  aircraft  and  ground  units.  The  use  of  JVMF  messaging  enhances  pilot’s  SA 
by  allowing  critical  infonnation  to  be  passed  quickly  without  the  pilot  having  to  resort  to 
conventional  voice  communication;  however,  it  also  requires  that  pilots  manage  incoming 
messages  and  create  outgoing  messages.  Although  JVMF  messaging  is  a  great  asset  to  pilots, 
there  are  concerns  with  the  workload  required  to  effectively  manage  the  large  amount  of  data 
created  by  such  a  large  flow  of  information. 

The  JVMF  messaging  was  the  subject  of  many  pilot  comments  during  the  LUT.  The  most 
common  issue  regarding  the  JVMF  messages  was  the  current  method  used  to  notify  pilots  of 
incoming  messages.  Currently,  the  JVMF  symbol  on  the  MFD  changes  to  inverse  video  to 
indicate  that  a  message  was  received.  Inverse  video  means  the  colors  of  the  text  and  background 
become  inverted.  The  pilots  stated  that  this  method  was  ineffective  and  difficult  to  detect.  As  a 
result,  many  messages  were  left  unread  until  the  pilots  realized  there  was  a  message. 

The  pilots  also  commented  on  two  workload-related  JVMF  issues.  First,  the  pilots  indicated  that 
sending  free-text  messages  was  workload  intensive  and  caused  them  to  remain  “heads  down”  for 
extended  periods  of  time.  The  second  issue  was  difficulty  navigating.  The  pilots  commented 
that  there  were  too  many  pages  associated  with  JVMF  messaging.  Several  of  the  pilots 
recommended  that  the  number  of  pages  be  reduced  and  the  remaining  pages  be  better  organized. 

Another  issue  the  pilots  identified  was  difficulty  in  using  the  backspace  function  on  the  FMS. 
The  JVMF  page  on  the  FMS  requires  the  use  of  the  left  arrow  button  to  backspace.  This  method 
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is  inconsistent  with  the  design  of  other  pages  in  the  FMS.  The  pilots  indicated  that  this  caused 
confusion  and  increased  their  workload  as  a  result. 


3.3.2  Collective  and  Cyclic  Flight  Controls 

The  pilots  were  asked  to  identify  problems  with  the  usability  of  the  collective  and  cyclic  flight 
controls  and  switches.  Two  issues  were  identified.  The  first  issue  was  that  the  frequency 
selection  switch  on  the  collective  was  not  intuitive.  The  pilots  indicated  that  this  switch  worked 
in  a  manner  opposite  from  that  expected.  They  recommended  that  the  design  of  the  switch  be 
improved  so  when  the  switch  is  pushed  “up,”  the  frequencies  scroll  “up”  and  vice  versa.  The 
second  issue  was  hand  interference  with  the  cyclic  mounted  stabilator  “slew-up”  switch  when 
pilots  wore  MOPP  gloves. 


3.4  Simulator  Sickness 

Pilots  reported  very  few  simulator  sickness  symptoms  during  the  LUT.  Most  of  the  symptoms 
involved  slight  sweating,  general  discomfort,  fatigue,  and  mild  eyestrain,  and  headache.  The 
overall  mean  total  severity  score  (post  mission)  for  the  pilots  was  7.49  (see  table  5).  The 
simulator  sickness  scores  were  also  compared  for  left  and  right  seat  positions.  The  total  severity 
scores  for  left  and  right  seats  were  5.58  and  9.33,  respectively.  These  results  show  a  large 
difference;  however,  the  difference  was  not  statistically  significant  (WSRT,  z  =  -0.944,/?  = 
0.345).  Overall,  the  S1L  posed  no  problems  for  simulator  sickness  and  should  continue  to  be 
very  suitable  as  a  simulation  environment  in  the  future. 


Table  5.  Simulator  sickness  questionnaire  ratings. 


Condition 

Nausea 

Subscale 

Oculomotor 

Subscale 

Disorientation 

Subseale 

Total  Severity 
Score  (Mean) 

SD 

Pre-Mission 

1.59 

1.06 

1.95 

1.67 

4.09 

Post  Mission 

6.36 

8.64 

2.71 

7.49 

6.12 

Right  seat 

4.75 

5.86 

3.09 

5.58 

8.47 

Left  seat 

7.93 

11.36 

2.32 

9.33 

7.41 

SD  =  standard  deviation 


3.4.1  Comparison  of  SIL  SSQ  Scores  to  Other  Helicopter  Simulators 

To  assess  whether  the  SSQ  ratings  provided  by  the  pilots  during  the  LUT  were  similar  to  or 
different  from  ratings  obtained  in  other  helicopter  simulators,  the  mean  total  severity  score  for 
the  SIL  was  compared  to  the  mean  total  severity  scores  for  several  other  helicopter  simulators 
(see  table  6):  the  AH-64A,  S-3H,  CH-46E,  CH-56D,  CH-56F,  Sikorsky  reconnaissance  attack 
helicopter  (RAH)-66  Engineering  Development  Simulator  (EDS),  RAH-66  Comanche  portable 
cockpit  (CPC),  and  the  Battlefield  Highly  Immersive  Virtual  Environment  (BHIVE).  These 
simulators  typically  induced  low  to  moderate  levels  of  simulator  sickness  symptoms  in  pilots. 
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Table  6.  Comparison  of  SIL  SSQ  ratings  with  other  helicopter  simulators. 


Simulator 

Nausea 

Subseale 

Oculomotor 

Subscale 

Disorientation 

Subscale 

Total  Severity  Score 
(Mean) 

AH-64A* 

25.81 

SH-3H 

14.70 

20.00 

12.40 

18.80 

RAH-66  EDS 

11.84 

14.98 

4.54 

13.25 

CH-53F 

7.50 

10.50 

7.40 

10.00 

RAH-66  CPC 

3.29 

12.94 

7.89 

9.80 

UH-60  BHIVE  (LEUE) 

6.36 

11.81 

3.09 

9.15 

UH-60M  BHIVE  (EUD) 

13.88 

6.89 

0 

8.50 

CH-53D 

7.20 

7.20 

4.00 

7.50 

SIL 

6.36 

8.64 

2.71 

7.49 

CH-46E 

5.40 

7.80 

4.50 

7.00 

*SSQ  subscale  data  not  available. 


3.5  Eye  Tracker 

The  importance  of  collecting  eye  tracker  data  was  to  determine  how  well  the  design  of  the 
aircraft  allowed  the  flying  pilot  to  remain  focused  outside  the  aircraft  during  visual  flight  rules 
(VFR).  Of  secondary  importance  was  the  percentage  of  time  the  non-flying  pilot  spent  focused 
outside  the  aircraft  during  VFR.  Analyzing  the  eye  tracker  data  to  answer  these  questions  was 
difficult  because  of  the  nature  of  this  operational  test.  First,  several  different  missions  were 
flown  that  have  an  impact  on  eye  tracker  data.  For  example,  VFR,  instrument  flight  rules  (IFR), 
single-ship,  and  multi-ship  missions  were  flown  during  the  LUT.  We  anticipated  differences  in 
the  eye  tracker  data  in  each  case,  so  we  evaluated  the  eye  tracker  data  collectively  and  separately. 
Another  challenge  was  that  the  pilots  were  allowed  to  transfer  the  flight  controls  whenever 
necessary,  as  is  the  customary  procedure  in  the  UH-60  aircraft.  In  order  to  correctly  analyze  the 
eye  tracker  data,  we  had  to  know  which  pilot  was  flying  at  all  times.  We  accomplished  this  by 
closely  monitoring  the  test  and  creating  new  data  segments  every  time  the  pilots  transferred  the 
controls.  These  segments  were  verified  post  mission  by  a  review  of  the  audio  and  video  tapes  of 
the  missions. 

Table  7  shows  the  percentage  of  time  the  flying  and  non- flying  pilots  spent  fixated  OTW. 

Notice  that  the  OTW  percentage  is  considerably  larger  for  both  the  flying  and  non-flying  pilots 
during  multi-ship  missions. 


Table  7.  OTW  eye  tracker  results. 


All  Missions 

Single-Ship  Missions 

Multi-Ship  Missions 

Flying  Pilot 

85.60% 

68.04% 

90.12% 

Non-Flying  Pilot 

28.21% 

23.50% 

32.96% 

Figure  13  shows  a  graphical  representation  of  the  eye  tracker  data  for  the  flying  pilot,  regardless 
of  seat  position.  This  figure  shows  the  percentage  of  mission  time  that  pilots  were  focused  on 
each  AOI.  The  data  for  the  flying  pilot  were  similar,  regardless  of  seat  position. 
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Figure  13.  Graphical  representation  of  eye  tracker  results  (flying  pilot). 


Figure  14  shows  a  graphical  representation  of  the  eye  tracker  data  for  the  non-flying  pilot, 
regardless  of  seat  position.  This  figure  shows  the  percentage  of  mission  time  that  pilots  were 
focused  on  each  AOI.  The  eye  tracker  data  for  the  non-flying  pilot  seemed  to  differ 
significantly,  depending  on  seat  position. 

A  complete  set  of  eye  tracker  data  showing  the  results  by  seat  position  and  mission  type  is 
presented  in  appendix  K. 
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Figure  14.  Graphical  representation  of  eye  tracker  results  (non-flying  pilot). 


3.5.1  Comparison  of  Eye  Tracker  Data  From  Previous  UH-60  Assessments 

The  LUT  was  one  of  three  evaluations  conducted  to  evaluate  the  suitability  of  the  UH-60M  crew 
station.  Eye  tracker  data  were  also  collected  during  two  other  UH-60  evaluations,  the  Early  User 
Demonstration  2  (EUD2)  and  the  Limited  Early  User  Evaluation  (LEUE).  Table  8  shows  a 
comparison  of  eye  tracker  data  from  each  of  the  three  evaluations.  Although  a  comparison  of  the 
results  of  each  evaluation  is  useful,  one  must  be  cautious  to  remember  that  each  set  of  data  was 
collected  in  different  evaluations  that  were  all  conducted  differently.  For  example,  the  scenarios 
flown  in  each  evaluation  were  different,  the  areas  of  interest  were  defined  differently  in  each 
evaluation,  and  a  different  number  of  participants  with  different  experience  levels  participated  in 
each  evaluation. 


Table  8.  Comparison  of  eye  tracker  results  from  EUD2,  LUT,  and  LEUE. 


EUD2 

LUT 

LEUE 

Flying  Pilot 

Non-Flying 

Pilot 

Flying  Pilot 

Non-Flying 

Pilot 

Flying  Pilot 

Non-Flying 

Pilot 

Outside 

69.21% 

N/A 

85.60% 

28.21% 

60.86% 

26.30% 

Inside 

30.79% 

N/A 

14.40% 

71.79% 

39.14% 

73.70% 
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4.  Summary 


4.1  Summary  of  Crew  Workload 

Pilots  reported  a  mean  workload  rating  of  2.7 1 .  This  indicates  that  there  was  enough  workload 
capacity  for  all  desirable  tasks. 

Two  tasks  received  peak  workload  ratings  of  greater  than  6.  A  workload  rating  higher  than  6 
indicates  that  workload  was  not  tolerable  for  the  task.  These  tasks  included  maintaining 
airspace  surveillance  and  transmitting  tactical  reports. 

The  TSC  rated  the  average  pilot  workload  at  2.91,  indicating  that  the  pilots  had  enough 
workload  capacity  for  all  desirable  tasks. 

Both  pilot  and  TSC  overall  workload  ratings  during  the  LUT  showed  an  improvement  over  the 
previous  UH-60M  assessment  during  EUD  2. 

4.2  Summary  of  Crew  Situational  Awareness 

Pilots  reported  a  mean  SA  rating  of  28.25  for  the  UH-60M — a  significant  increase  compared  to 
their  SA  rating  of  1 1 .69  for  the  UH-60A/L. 

The  SA  ratings  for  the  left  and  right  seats  were  statistically  insignificant. 

Increased  SA  was  attributed  to  the  addition  of  the  digital  map  system. 

The  tactical  steering  committee  rated  crew  SA  as  1.47.  This  indicates  that  the  crew  was  aware 
of  the  battlefield  and  their  own  ship  with  minor  or  insignificant  variation  between 
perception  and  reality. 

Both  pilot  and  TSC  overall  SA  ratings  during  the  LUT  showed  an  improvement  over  the 
previous  UH-60M  assessment  during  EUD  2. 

4.3  PVI 

The  pilots  indicated  that  the  “inverse  video”  JVMF  notification  was  difficult  to  detect.  As  a 
result,  many  messages  were  left  unread  until  pilots  realized  there  was  a  message. 

The  task  of  sending  free-text  messages  was  workload  intensive  and  caused  pilots  to  remain 
“heads  down”  for  extended  periods. 

The  pilots  indicated  that  too  many  pages  were  associated  with  JVMF  messaging,  which  caused 
difficulty  in  navigating  the  menu  structure. 
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The  pilots  identified  an  inconsistency  with  the  function  of  the  backspace  function  on  the  FMS. 
The  backspace  function  was  associated  with  different  keys  on  the  FMS,  depending  on 
which  page  was  being  used. 

The  frequency  select  switch,  located  on  the  collective,  was  not  intuitive. 

The  pilots  experienced  hand  interference  from  the  cyclic  mounted  stabilator  slew-up  switch 
when  they  wore  MOPP  gloves. 

4.4  Simulator  Sickness 

Pilots  reported  mild  simulator  sickness  symptoms  after  flying  missions  in  the  SIL.  The  total 
severity  score  was  7.49  with  higher  scores  from  the  right  seat. 

Simulator  sickness  symptoms  did  not  adversely  affect  pilot  performance. 

4.5  Eye  Tracker 

The  flying  pilot  spent  85.60%  of  the  time  fixated  OTW,  4.93%  fixated  on  the  left  MFD,  and 
5.93%  on  the  right  MFD.  The  remaining  time  was  spent  looking  at  other  areas  that  were 
not  specifically  examined  during  this  evaluation  (e.g.,  FMS,  kneeboard,  center  console, 
etc.) 

The  non-flying  pilot  spent  28.21%  fixated  OTW,  21.11%  on  the  left  MFD,  and  26.40%  on  the 
right  MFD.  The  remaining  time  was  spent  looking  at  other  areas  that  were  not  specifically 
examined  during  this  evaluation  (e.g.,  FMS,  kneeboard,  center  console,  etc.) 

A  large  difference  was  noted  between  OTW  percentages  between  single  and  multi-ship 

missions.  The  average  flying  pilot  OTW  percentage  for  single-ship  missions  was  68.04%, 
and  the  average  flying  pilot  OTW  percentage  for  multi-ship  missions  was  90.12%.  This 
difference  was  attributed  to  the  fact  that  pilots  typically  focus  outside  more  during  multi¬ 
ship  missions  to  maintain  their  position  relative  to  the  aircraft  in  front  of  them. 

The  eye  tracker  results  from  the  LUT  showed  an  improvement  in  OTW  percentages  for  the 
flying  pilot  compared  to  the  previous  UH-60M  assessment  during  EUD  2. 


5.  Recommendations 


The  following  recommendations  are  made  to  enhance  the  overall  effectiveness  and  suitability  of 
the  UH-60M  as  it  continues  its  development: 

Address  and  resolve  the  workload  and  PVI  issues  identified  during  the  LUT. 
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Use  the  Crew  Station  Working  Group,  MANPRINT  Working  Group,  and  System  Safety 
Working  Group  to  track  issues  until  satisfactorily  resolved. 


Continue  to  assess  the  crew  station  during  future  simulations  and  tests  to  evaluate  pilot  and 
system  performance  and  assess  new  functionality  that  is  integrated  into  the  UH-60M  design. 

Data  from  the  workload,  SA,  and  SSQ,  plus  the  data  from  the  eye  tracker,  should  be  collected 
again  during  future  UH-60M  crew  station  evaluations.  This  procedural  continuity  will  allow 
direct  comparison  after  further  design  and  development  of  the  UH-60M  crew  station  to  check  for 
continued  improvements  in  workload,  SA,  and  PVI. 
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Appendix  A.  Bedford  Workload  Rating  Scale 


1.  PIN _  2.  Date  (DD/MMM/YY): _ / _ /04 

3.  Mission  ID  number  _ 

4.  Right  Seat _  Left  Seat  _  (Check  one) 


Workload 


5.  Rate  the  workload  for  the  Flight  and  Mission  Tasks  you  perfonned  in  comparison  to  your 
experiences  in  the  UH-60A/L.  Use  the  scale  provided  on  the  last  page  of  this  questionnaire.  For 
example,  if  on  Task  1026  (Perform  Electronically  Aided  Navigation)  you  nonnally  feel  your 
workload  would  be  a  ‘5’,  indicate  that  in  the  column  under  UH-60A/L.  With  that  in  mind,  make 
a  workload  judgment  for  Task  1026  in  the  UH-60M.  Place  the  workload  rating  in  the  blank  next 
to  each  Flight  and  Mission  Task.  If  you  did  not  perform  a  task  during  the  mission  that  you  just 
completed,  place  an  X  in  the  non-applicable  (N/A)  column. 


Task 

No. 

Flight  and  Mission  Tasks 

UH-60A/L 

Workload 

UH-60M 

Workload 

N/A 

1014 

Maintain  Airspace  Surveillance 

1016 

Perform  Hover  Power  Check 

1017 

Perform  Hovering  Flight 

1018 

Perform  VMC  Takeoff 

1023 

Perform  Fuel  Management  Procedures 

1025 

Navigate  by  Pilotage  and  Dead  Reckoning 

1026 

Perform  Electronically  Aided  Navigation 

1028 

Perform  VMC  Approach 

1029 

Perform  a  Roll-on  Landing 

1068 

Perform  Emergency  Procedures 

1076 

Perform  Radio  Navigation 

1077 

Perform  Holding  Procedures 

1079 

Perform  Radio  Communication  Procedures 

1081 

Perform  Nonprecision  Approach 

1082 

Perform  Precision  Approach 

1083 

Perform  Inadvertent  IMC  Procedures 

1084 

Perform  Command  Instrument  System  Operations 

1095 

Operate  Aircraft  Survivability  Equipment 

1135 

Perform  Instalment  Maneuvers 

1136 

Perform  Go-Around 

1146 

Perform  VMC  Flight  Maneuvers 

1150 

Select  Landing  Zone/Pickup  Zone 

2008 

Perform  Evasive  Maneuvers 

2009 

Perform  Multi-Aircraft  Operations 
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Task 

No. 

Flight  and  Mission  Tasks  (cont’d) 

UH-60A/L 

Workload 

UH-60M 

Workload 

N/A 

2044 

Perform  Actions  on  Contact 

2078 

Perform  Terrain  Flight  Mission  Planning 

2079 

Perform  Terrain  Flight  Navigation 

2081 

Perform  Terrain  Flight 

2083 

Negotiate  Wire  Obstacles 

2086 

Perform  Masking  and  Unmasking 

2090 

Perform  Tactical  Communication  Procedures 

2091 

Transmit  Tactical  Reports 

If  you  gave  a  workload  rating  of  ‘6’  or  higher  for  any  task  on  the  UH-60M  only,  explain  why  the 
workload  was  high  for  the  task. 


In  the  mission  you  just  flew,  list  any  flight  and/or  mission  tasks  on  the  UH-60M  that  you  had  to 
ask  your  crewmember  to  accomplish  because  your  workload  was  too  high: 
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Workload  Description  “Rating” 


YES 


— * 

Wortdoad  insignificant 

1 

Workload  low 

2 

— ► 

Enough  spare  capacity  for  all 
desirable  additional  tasks 

3 

Was  wa&kM 

saiie&steisx 

wit.ljgjit  reduction  in 
sp-^  (workload)  capacity? 


NO 


Task  abandoned.  Pilot  unable  to 
apply  sufficient  effort 


Insufficient  spare  capacity  for  easy 
attention  to  additional  tasks 

4 

Reduced  spare  capacity.  Additional 
tasks  cannot  be  given  the  desired 
amount  of  attention 

5 

Little  spare  capacity:  level  of 
allows  little  attention  to  additional 
tasks 

6 

Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 

7 

Very  lugihvrarkbadvath  almost  no 

- ► 

Difficulty  in 

maintaining  level  of  effort 

8 

Extremely  high  workload.  No  spare 
capacity.  Serious  doubts  as  to  ability 
to  maintain  level  of  effort 

9 

10 
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Intentionally  left  blank 
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Appendix  B.  Situational  Awareness  Rating  Technique 


Pin  # _  Date  (DD/MM/YY): _ / _ /  04 

Mission  ID  Number:  _ 

Right  Seat _  Left  Seat  _  (Check  one) 

Situation  Awareness 


SA1.  Situation  Awareness  is  defined  as  “timely  knowledge  of  what  is  happening  as  you  perform 
your  right  or  left  seat  tasks  during  the  mission.” 


Situation  Awareness  Rating  Technique  (SART) 

DEMAND 

Instability  of  Situation 

Likeliness  of  situation  to  change  suddenly. 

Variability  of  Situation 

Number  of  variables  which  require  your  attention 

Complexity  of  Situation 

Degree  of  complication  (number  of  closely  connected  parts)  of  the  situation 

SUPPLY 

Arousal 

Degree  to  which  you  are  ready  for  activity;  ability  to  anticipate  and  keep  up  with 
the  flow  of  events 

Spare  Mental  Capacity 

Amount  of  mental  ability  available  to  apply  to  new  tasks 

Concentration 

Degree  to  which  your  thoughts  are  brought  to  bear  on  the  situation;  degree  to 
which  you  focused  on  important  elements  and  events 

Division  of  Attention 

Ability  to  divide  your  attention  among  several  key  issues  during  the  mission; 
ability  to  concern  yourself  with  many  aspects  of  current  and  future  events 
simultaneously 

UNDERSTANDING 

Information  Quantity 

Amount  of  knowledge  received  and  understood 

Information  Quality 

Degree  of  goodness  or  value  of  knowledge  communicated 

Familiarity 

Degree  of  acquaintance  with  the  situation 
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Assuming  you  had  just  performed  this  mission  in  a  UH-60A/L,  rate  the  level  of  each  component 
of  situation  awareness  that  you  had.  Circle  the  appropriate  number  for  each  component  of 
situation  awareness  (e.g.,  complexity  of  situation). 


Instability  of  situation:  Low 
Variability  of  situation:  Low 
Complexity  of  situation:  Low 


DEMAND 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 


SUPPLY 


Arousal: 

Low 

Low 

1 - 

i 

— 2- 

_ _ 2— 

- 3. 

3 

4  - 

- — 5- 

s 

_ 5 _ 

- 7 

7 

High 

High 

Concentration: 

Low 

1 

_ 2— 

3 

4 

_ 5 _ 

7 

High 

Division  of  attention: 

Low 

1 - 

-2- 

- 3. 

_ 4 _ 

- — 5- 

_ 6 _ 

- 7 

High 

UNDERSTANDING 

Information  quantity: 

Low 

1 _ 2 _ 3 _ 4 _ 

_ 5 _ 

_ 6 _ 

7 

High 

Information  quality: 

Low 

1 _ 2 _ 3 _ 4 _ 

_ 5 _ 

_ 6 _ 

7 

High 

Familiarity: 

Low 

1 _ 2 _ 3 _ 4 _ 

_ 5 _ 

. — 6-— 

. — 7 

High 
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For  the  mission  that  you  just  completed  in  the  UH-60M,  rate  the  level  of  each  component  of 
situation  awareness  that  you  had.  Circle  the  appropriate  number  for  each  component  of  situation 
awareness  (e.g.,  complexity  of  situation). 


Instability  of  situation:  Low 
Variability  of  situation:  Low 
Complexity  of  situation:  Low 


DEMAND 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 


Arousal: 

Spare  mental  capacity: 
Concentration: 
Division  of  attention: 


SUPPLY 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 

Low  1 - 2 - 3 - 4 - 5 - 6 - 7  High 


UNDERSTANDING 

Information  quantity: 

Low 

1 _ 2 _ 3 _ 4 _ 

_ 5 _ 

_ 6 _ 

7 

High 

Information  quality: 

Low 

1 _ 2 _ 3 _ 4 _ 

_ 5 _ 

_ 6 _ 

7 

High 

Familiarity: 

Low 

1 _ 2 _ 3 _ 4 _ 

_ 5 _ 

. — 6-— 

. — 7 

High 
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SA2.  Rate  the  level  of  situational  awareness  you  had  for  each  of  the  battlefield  elements  during 
the  mission  by  placing  and  X  in  the  appropriate  column  for  each  battlefield  element.  Keep  in 
mind  that  the  simulation  facility  may  be  limited  in  its  ability  to  display  some  of  these  elements; 
in  the  case,  please  place  ‘N/A’  somewhere  in  the  row  for  that  battlefield  element. 

Very  High  Fairly  High  Intermediate  Fairly  Low  Very  Low 

Battlefield  Level  of  Level  of  Level  of  Level  of  Level  of 

Elements  Situation  Situation  Situation  Situation  Situation 

Awareness  Awareness  Awareness  Awareness  Awareness 


Location  of 
Enemy  Units 

Location  of 
Friendly  Units 

Location  of 
Non- 

Combatants 
(e.g.,  Civilians) 

Location  of  My 
Aircraft  During 
Mission 

Location  of 
Other  Aircraft 
Related  to  the 
Mission 

Location  of 
Cultural 
Features  (e.g., 
bridges) 

Route 

Information 
(ACPs,  BPs, 
EAs,  RPs,  etc.) 

Status  of  My 
Aircraft 
Systems  (e.g., 
fuel 


Describe  any  instances  when  you  feel  you  had  low  situational  awareness  during  the  mission: 
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Intentionally  left  blank 
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Appendix  C.  PVI  Questionnaire 


1.  PIN _  2.  Date  (DD/MMM/YY): _ / _ /04 

3.  Mission  ID  number  _ 

4.  Right  Seat _  Left  Seat  _  (Check  one) 

The  purpose  of  this  questionnaire  is  to  identify  any  problems  that  you  experienced  when  using 
the  various  crew  station  components  to  perfonn  your  mission  tasks.  Your  responses  should  be 
based  only  on  the  problems  that  you  experienced  during  the  mission  that  you  just  completed. 

PVI.  The  following  table  lists  the  functional  components  (and  some  sub-components)  of  the 
UH-60M  crew  station.  For  each  functional  component  (and  sub-component),  indicate  whether  or 
not  you  experience  a  problem  using  the  component  in  a  quick  and  efficient  manner  during  the 
mission  you  just  completed.  Check  “Yes”  if  you  experience  one  or  more  problems.  Check  “No” 
if  you  did  not  experience  any  problems.  Check  “Not  Used”  if  you  did  not  use  the  functional 
component  during  the  mission  you  just  completed. 


•  Multifunction  Displays  (MFD) 

Yes 

No 

_  Not  Used  _ 

o  Primary  Flight  Display  (PFD) 

Yes 

No 

_  Not  Used  _ 

o  Navigation  Display  (ND) 

Yes 

No 

_  Not  Used  _ 

o  Engine  Instrument  Caution 
Advisory  System  (EICAS) 

Yes 

No 

_  Not  Used  _ 

o  Digital  Map  System  (DMS) 

Yes 

No 

_  Not  Used  _ 

o  Joint  Variable  Message  Format 
(JVMF) 

Yes 

No 

_  Not  Used  _ 

•  Flight  Management  System  (FMS) 

Yes 

No 

_  Not  Used  _ 

o  JVMF  Entry 

Yes 

No 

_  Not  Used  _ 

o  GPS  /  Flight  Plan 

Yes 

No 

_  Not  Used  _ 

o  Voice  Communications 

Yes 

No 

_  Not  Used  _ 

o  Radio  Navigation 

Yes 

No 

_  Not  Used  _ 

•  Multifunction  Slew  Controller  (MFSC) 

Yes 

No 

Not  Used  _ 
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•  Flight  Director  Display  Control  Panel 

(FDDCP)  Yes _  No _  Not  Used _ 

If  you  answered  “Yes”  to  any  of  the  questions,  please  describe  a)  the  problems  you  experienced, 
b)  how  much  the  problems  degraded  your  performance,  and  c)  any  recommendation  you  have  for 
improving  the  design  of  the  various  functional  components. 
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PV2.  On  average,  how  quickly  were  you  able  to  navigate  through  menu  screens  on  the: 

Primary  Flight  /  Navigation  Displays  (PFD/ND)  (Circle  one) 
12  3  45 


Very  Somewhat  Borderline  Somewhat  Very 

Quickly  Quickly  Slowly  Slowly 


Digital  Map  System  (DMS)  (Circle  one) 

12  3  45 


Very  Somewhat  Borderline  Somewhat  Very 

Quickly  Quickly  Slowly  Slowly 


Flight  Management  System  (FMS)  (Circle  one) 
12  3  45 


Very  Somewhat  Borderline  Somewhat  Very 

Quickly  Quickly  Slowly  Slowly 


Engine  Instrument  Caution  Advisory  System  (EICAS)  (Circle  one) 
12  3  45 


Very  Somewhat  Borderline  Somewhat  Very 

Quickly  Quickly  Slowly  Slowly 


Joint  Variable  Message  Format  (JVMF) 

12  3  45 


Very  Somewhat  Borderline  Somewhat  Very 

Quickly  Quickly  Slowly  Slowly 


If  you  answered  “Borderline”,  “Somewhat  Slowly”,  or  “Very  Slowly”  to  any  of  the  questions, 
list  the  component  and  why  navigation  was  slow  (e.g.,  ‘navigating  the  menu  system  on  the  FMS 
was  a  slow  process  due  to  having  to  page  through  several  screen  displays’). 
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PV3.  How  often  did  you  forget  the  steps  required  for  navigating  through  the  menu  screens  to 
accomplish  a  task? 

Primary  Flight  /  Navigation  Displays  (PFD/ND)  (Circle  one) 

12  3  4 

Never  Seldom  Frequently  Always 

Digital  Map  System  (DMS)  (Circle  one) 

12  3  4 

Never  Seldom  Frequently  Always 

Flight  Management  System  (FMS)  (Circle  one) 

12  3  4 

Never  Seldom  Frequently  Always 


Engine  Instrument  Caution  Advisory  System  (EICAS)  (Circle  one) 
12  3  4 


Never  Seldom  Frequently  Always 

Joint  Variable  Message  Format  (JVMF) 

12  3  4 


Never  Seldom  Frequently  Always 
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If  you  answered  “Frequently”  or  “Always”  to  any  of  the  questions,  list  the  component  and  the 
tasks  for  which  you  forgot  how  to  navigate  through  the  menu  screens  (e.g.,  “I  often  forgot  the 
steps  for  navigating  through  the  menu  screens  on  the  FMS  to  change  frequencies  on  the  UHF 
radio”). 
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PV4.  Please  rate  the  intuitiveness  of  the  following  aspects  of  the  Multifunction  Switch  Controller 
(MFSC)  (a.k.a.  potato  grip): 

PV4-1.  When  you  actuated  the  directional  control,  did  the  cursor  move  in  the  direction  you 
expected? 

12  3  45 


Very  Somewhat  Neither  Intuitive  Somewhat  Very 

Intuitive  Intuitive  nor  Confusing  Confusing  Confusing 


PV4-2.  When  you  actuated  one  of  the  three  switches,  did  the  expected  action  occur  on  the  MFD? 
1  2  3  4  5 


Very  Somewhat  Neither  Intuitive  Somewhat  Very 

Intuitive  Intuitive  nor  Confusing  Confusing  Confusing 

If  you  answered  “Neither  Intuitive  nor  Confusing”,  “Somewhat  Confusing”,  or  “Very 
Confusing”,  please  describe  any  problem  with  either  the  cursor  control  or  switches,  exactly  what 
you  were  trying  to  accomplish  on  the  MFD,  and  what  actually  happened  on  the  MFD. 
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PV5.  Did  you  have  any  difficulty  using  any  of  the  switches  on  the  collective  or  the  cyclic  grips? 

Collective  Grip  Yes _  No _ 

Cyclic  Grip  Yes _  No _ 

If  you  answered  “Yes”  for  either  flight  control,  please  list  which  flight  control  and  switch(es), 
and  the  problems  you  experienced  (e.g.,  confuses  two  switches  due  to  similar  shape,  switch  too 
hard  to  reach). 
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PV6.  Was  there  any  symbology  depicted  on  the  following  displays/pages  that  was  difficult  to 
quickly  and  easily  understand? 


Primary  Flight  Displays  (PFD) 

Yes 

No 

Navigation  Displays  (ND) 

Yes 

No 

EICAS 

Yes 

No 

Digital  Map  System  (DMS) 

Yes 

No 

Aircraft  Survivability  Equip  (ASE) 

Yes 

No 

If  you  answered  “Yes”  to  any  of  the  questions,  please  describe  a)  the  display/page,  b)  the 
symbology  that  was  difficult  understand,  c)  how  the  symbology  may  have  degraded  your 
performance,  and  d)  any  recommendation  you  have  for  improving  the  design  of  the  various 
functional  components. 
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PV7.  Did  you  experience  any  problems  with  symbology  cluttter  on  the  following  displays  that 
made  it  difficult  to  understand  all  the  elements  of  information  available  to  you? 


PFD  -  Full 

Yes 

No  _ 

PFD  -  Arc 

Yes 

No  _ 

PFD  -  Hover 

Yes 

No  _ 

ND  -  Full 

Yes 

No  _ 

ND  -  Plan 

Yes 

No  _ 

Digital  Map 

Yes 

No  _ 

If  you  answered  “Yes”  for  any  of  these  displays,  please  indicate  which  display  and  what  symbols 
were  cluttering  the  display  to  make  it  difficult  to  understand.  Please  include  any 
recommendation  you  might  have  to  alleviate  the  difficulty. 
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PV8.  Did  you  experience  any  sort  of  hand  discomfort  while  using  the  MSFC,  collective,  or 
cyclic  grips? 


MFSC 

Yes 

No  _ 

Collective 

Yes 

No  _ 

Cyclic 

Yes 

No  _ 

If  you  answered  “Yes”  for  any  of  these  controls,  please  list  which  control  became  uncomfortable, 
a  rough  description  of  how  your  hand  was  uncomfortable,  what  tasks  you  were  trying  to 
accomplish,  and  approximately  how  long  it  took  for  your  hand  to  become  uncomfortable. 
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PV9.  For  the  JVMF  reports  that  you  sent,  how  would  you  rate  the  ease/difficulty  of  sending  the 
following  reports: 


Position  Report 

12  3  45 


Very  Somewhat 

Borderline 

Somewhat 

Very 

Easy  Easy 

Difficult 

Difficult 

Free  Text  Message 

1  2 

3 

4 

5 

Very  Somewhat  Borderline  Somewhat  Very 

Easy  Easy  Difficult  Difficult 


If  you  answered  “Borderline”,  “Somewhat  Difficult”,  or  “Very  Difficult”,  please  indicate  which 
type  of  message  you  sent,  the  exact  difficulties  you  encountered,  and  any  recommendations  to 
alleviate  the  problem. 
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PV10.  How  would  you  rate  your  ability  to  detect  the  following  occurrences  based  on  the 
characteristics  of  the  flight  displays? 

JVMF  Message  (MFD) 

12  3  45 


Very  Somewhat  Borderline  Somewhat  Very 

Easy  Easy  Difficult  Difficult 

Caution  /  Advisory  (MFD) 

1  2  3  4  5 


Very  Somewhat  Borderline 

Easy  Easy 

Warning  (Master  Warning  Panel) 

1  2  3 


Somewhat  Very 

Difficult  Difficult 


4  5 


Very  Somewhat  Borderline  Somewhat  Very 

Easy  Easy  Difficult  Difficult 

Entry  into  Operational  Limits  (per  Chp  5)  on  the  Power  Pod 

12  3  45 


Very  Somewhat  Borderline  Somewhat  Very 

Easy  Easy  Difficult  Difficult 

If  you  answered  “Borderline”,  “Somewhat  Difficult”,  or  “Very  Difficult”,  please  indicate  which 
annunciation  you  had  difficulty  detecting,  why  you  may  have  had  difficulty  detecting  it,  and  any 
recommendations  to  make  the  annunciation  more  noticeable  or  salient. 


48 


Appendix  D.  Simulator  Sickness  Questionnaire 


1.  PIN#: _  2.  Date  (DD/MMM/YY): _ - _ -04 

3.  Mission  ID  Number:  _ 

4.  Seat  you  will  fly  from:  Right  Seat _  Left  Seat  _  (Check  one) 

5.  Please  indicate  the  severity  of  symptoms  that  apply  to  you  right  now  by  circling  the  appropriate  word. 


Symptom 

0 

1 

2 

3 

a. 

General  discomfort 

None 

Slight 

Moderate 

Severe 

b. 

Fatigue 

None 

Slight 

Moderate 

Severe 

c. 

Headache 

None 

Slight 

Moderate 

Severe 

d. 

Eyestrain 

None 

Slight 

Moderate 

Severe 

e. 

Difficulty  focusing 

None 

Slight 

Moderate 

Severe 

f. 

Increased  salivation 

None 

Slight 

Moderate 

Severe 

g- 

Sweating 

None 

Slight 

Moderate 

Severe 

h. 

Nausea 

None 

Slight 

Moderate 

Severe 

i. 

Difficulty  concentrating 

None 

Slight 

Moderate 

Severe 

j- 

Fullness  of  head 

None 

Slight 

Moderate 

Severe 

k. 

Blurred  vision 

None 

Slight 

Moderate 

Severe 

1.  Dizzy  (eyes  open) 

None 

Slight 

Moderate 

Severe 

m. 

.  Dizzy  (eyes  closed) 

None 

Slight 

Moderate 

Severe 

n. 

* 

Vertigo 

None 

Slight 

Moderate 

Severe 

0. 

** 

Stomach  awareness 

None 

Slight 

Moderate 

Severe 

P- 

Burping 

None 

Slight 

Moderate 

Severe 

Vertigo  is  a  loss  of  orientation  with  respect  to  vertical  upright. 


Stomach  awareness  is  a  feeling  of  discomfort  just  short  of  nausea. 


6.  Are  you  in  your  usual  state  of  health  and  fitness? 

YES 

NO 

7a.  Have  you  been  ill  in  the  past  week? 

YES 

NO 

b.  If  yes,  are  you  fully  recovered? 

YES 

NO 

N/A 
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1.  PIN#: 


2.  Date  (DD/MMM/YY): 


-04 


3.  Mission  ID  Number: 

4.  Seat  you  flew  from:  Right  Seat 

Left  Seat 

(Check  one) 

5.  Please  indicate  the  severity  of  symptoms  that  apply  to  you  risht 

now  by  circling  the  appropriate  word. 

Symptom 

0 

1 

2 

3 

a.  General  discomfort 

None 

Slight 

Moderate 

Severe 

b.  Fatigue 

None 

Slight 

Moderate 

Severe 

c.  Headache 

None 

Slight 

Moderate 

Severe 

d.  Eyestrain 

None 

Slight 

Moderate 

Severe 

e.  Difficulty  focusing 

None 

Slight 

Moderate 

Severe 

f.  Increased  salivation  None 

Slight 

Moderate  Severe 

g.  Sweating 

None 

Slight 

Moderate 

Severe 

h.  Nausea 

None 

Slight 

Moderate 

Severe 

i.  Difficulty  concentrating 

None 

Slight 

Moderate 

Severe 

j.  Fullness  of  head 

None 

Slight 

Moderate 

Severe 

k.  Blurred  vision 

None 

Slight 

Moderate 

Severe 

1.  Dizzy  (eyes  open) 

None 

Slight 

Moderate 

Severe 

m.  Dizzy  (eyes  closed)  None 

Slight 

Moderate  Severe 

n.  Vertigo* 

None 

Slight 

Moderate 

Severe 

o.  Stomach  awareness**  None 

Slight 

Moderate  Severe 

p.  Burping  None  Slight  Moderate  Severe 

Vertigo  is  a  loss  of  orientation  with  respect  to  vertical  upright. 

Stomach  awareness  is  a  feeling  of  discomfort  just  short  of  nausea. 
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Appendix  E.  TSC  Survey 


Pin:  _ 

Mission  Trial _  Date  (DD/MMM/YY): _ / _ /04 

TSCWL1.  Place  the  workload  rating  in  the  blank  next  to  each  crewmember  using 
the  rating  scale  on  the  next  page. 


Crew  Members 

Overall  Workload  Rating  For 
This  Mission 

Left  Seat 

Right  Seat 

If  you  assigned  a  workload  rating  of  ‘6’  or  higher  for  either  crewmember,  explain  why: 


TSCWL2.  Rate  the  effectiveness  of  aircrew  coordination  as  defined  by  the  USAAVNC  Aircrew 
Coordination  ETP  and  TC  1-210. 

1  2  3  4  5 


Excellent  Good  Average  Needs  Improvement  Unacceptable 
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Workload  Description  “Rating;'' 


YES 


— ► 

Workload  insignificant 

1 

Woikload  low 

2 

— ► 

Enough  spare  capacity  for  all 
desirable  additional  tasks 

3 

Was  TjfpifElpad. 
reduction  in 

sp$*$  (workload)  capacity? 


NO 


Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 

spare.Cj(pacit.Y.  Difficulty  in 
maintaining  level  of  effort 

No  spare 

capacity.  Serious  doubts  as  to  ability 
to  maintain  level  of  effort 


Task  abandoned.  Pilot  unable  to 
apply  sufficient  effort 


Insufficient  spare  capacity  for  easy 
attention  to  additional  tasks 

4 

Reduced  spare  capacity.  Additional 
tasks  cannot  be  given  the  desired 
amount  of  attention 

5 

Little  spare  capacity:  level  of  gffgjj:,,, 
allows  little  attention  to  additional 
tasks 

6 

10 
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PIN 


Date  (DD/MMM/YY): 


/ 


/  0  4 


Mission  ID  number 


TSC  SITUATION  AWARENESS  RATING  SCALE 


Check  one 

Crew  was  consistently  aware  of  all  entities  on  the  battlefield  as  well  as  the 

status  of  their  aircraft 

Crew  was  aware  of  the  battlefield  and  their  own  ship  with  minor  or 
insignificant  variation  between  perception  and  reality. 

Crew  was  aware  of  the  battlefield  and  their  own  ship.  Variation  between 
reality  and  perception  did  not  significantly  impact  mission  success. 

SA  needs  improvement.  Lack  of  SA  had  some  negative  effect  on  the  success 

of  the  mission. 

Lack  of  SA  caused  mission  failure. 

Describe  any  problems  that  aircrews  had  with  situation  awareness. 


PIN 


Date  (DD/MMM/YY): _ / _ /04 
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Mission  ID  number 


TSC  MISSION  SUCCESS  QUESTIONNAIRE 

TSC  MSI.  Did  the  UH-60M  crew  complete  their  mission  objectives? 

Yes  _  No  _ 

If  no,  why  weren’t  the  mission  objectives  completed? 


TSC  MS2.  Was  the  mission  successful? 

Yes  _  No 

If  no,  what  caused  the  mission  to  fail? 
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Appendix  F.  Mean  Workload  Ratings  for  All  ATM  Tasks 


Mean  Workload  Ratings  for  All  ATM  Tasks 

Task 

No. 

Task  Description 

101 

105 

140 

192 

213 

645 

Average 

1014 

Mantain  Airspace  Surveillance 

1.83 

4.00 

3.00 

3.00 

2.50 

1.83 

2.69 

1016 

Perform  Hover  Power  Check 

1.83 

2.83 

3.00 

4.00 

2.33 

1.67 

2.61 

1017 

Perform  Hovering  Flight 

1.83 

2.67 

2.83 

4.40 

2.17 

1.67 

2.59 

1018 

Perform  VMC  Takeoff 

1.83 

3.33 

3.00 

4.80 

2.50 

1.67 

2.86 

1023 

Perform  Fuel  Management  Procedures 

2.00 

3.17 

3.00 

4.80 

2.17 

2.00 

2.86 

1025 

Navigate  by  Pilotage  and  Dead  Reckoning 

1.83 

1.75 

2.00 

3.00 

2.17 

2.00 

2.13 

1026 

Perform  Electronically  Aided  Navigation 

1.67 

2.33 

2.00 

3.75 

2.17 

1.83 

2.29 

1028 

Perform  VMC  Approach 

1.83 

2.83 

3.00 

5.20 

2.50 

2.00 

2.89 

1029 

Perform  a  Roll-on  Landing 

1.67 

2.75 

3.00 

5.00 

3.00 

2.33 

2.96 

1068 

Perform  Emergency  Procedures 

1.75 

3.40 

3.00 

4.67 

2.67 

2.00 

2.91 

1076 

Perform  Radio  Navigation 

1.50 

2.50 

3.00 

4.00 

2.00 

- 

2.60 

1077 

Perform  Holding  Procedures 

1.50 

2.50 

3.00 

5.50 

- 

2.00 

2.90 

1079 

Perform  Radio  Communication  Procedures 

1.83 

3.00 

3.00 

5.20 

2.17 

- 

3.04 

1081 

Perform  Nonprecision  Approach 

1.00 

1.00 

3.00 

- 

2.00 

1.50 

1.70 

1082 

Perform  Precision  Approach 

1.50 

2.50 

3.00 

4.00 

2.00 

3.50 

2.75 

1083 

Perform  Inadvertent  IMC  Procedures 

2.00 

3.00 

3.50 

5.00 

3.50 

- 

3.40 

1084 

Perform  Command  Instrument  System  Operations 

1.00 

2.00 

- 

5.00 

3.00 

- 

2.75 

1095 

Operate  Aircraft  Survivability  Equipment 

1.00 

- 

- 

5.00 

2.00 

2.50 

2.63 

1135 

Perform  Instrument  Maneuvers 

1.50 

2.50 

3.00 

5.00 

2.50 

- 

2.90 

1136 

Perform  Go-Around 

1.50 

1.00 

- 

- 

- 

2.00 

1.50 

1146 

Perform  VMC  Flight  Maneuvers 

1.67 

2.83 

3.00 

5.20 

2.67 

2.00 

2.89 

1150 

Select  Landing  Zone/Pickup  Zone 

1.83 

3.17 

2.75 

4.00 

2.33 

2.00 

2.68 

2008 

Perform  Evasive  Maneuvers 

1.50 

1.00 

3.00 

5.33 

2.50 

2.00 

2.56 

2009 

Perform  Multi- Aircraft  Operations 

2.67 

3.00 

3.00 

5.25 

2.67 

2.00 

3.10 

2044 

Perform  Actions  on  Contact 

2.00 

1.00 

3.00 

5.00 

2.00 

- 

2.60 

2078 

Perform  Terrain  Flight  Mission  Planning 

3.00 

3.33 

2.67 

- 

2.60 

2.17 

2.75 

2079 

Perform  Terrain  Flight  Navigation 

1.50 

3.00 

2.20 

3.40 

2.17 

2.00 

2.38 

2081 

Perform  Terrain  Flight 

1.83 

3.33 

2.67 

4.40 

2.50 

2.00 

2.79 

2083 

Negotiate  Wire  Obstacles 

1.83 

3.00 

2.80 

5.25 

2.33 

- 

3.04 

2086 

Perform  Masking  and  Unmasking 

- 

1.00 

3.00 

4.00 

- 

2.20 

2.55 

2090 

Perform  Tactical  Communication  Procedures 

2.60 

3.50 

3.00 

5.20 

2.60 

2.00 

3.15 

2091 

Transmit  Tactical  Reports 

2.60 

3.83 

3.20 

4.67 

2.50 

- 

3.36 
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Intentionally  left  blank 
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Appendix  G.  Pilot  Workload  Comments 


Task  (1014) 

•  JVMF  messaging.  Checking  aircraft  systems.  Fuel  Checks. 

•  One  pilot  would  be  responsible  for  radios  and  navigation.  The  other  pilot  would  fly  the 
aircraft. 

•  Pilot  on  the  controls  stays  outside  when  VMC,  inside  when  IMC.  Pilot  not  on  the 
controls  announced  when  inside  more  than  2-3  seconds  when  VMC,  and  scanned 
instruments  when  IMC.  If  pilot  came  off  instrument  scan  more  than  2-3  seconds  during 
IMC  the  pilot  announced  it. 

Task  (1016) 

•  One  pilot  would  fly  the  other  would  check  the  systems. 

•  Pilot  on  the  controls  announced  when  stable  at  desired  hover  height.  Pilot  announced 
hover  torque,  compared  it  to  predicted  and  go  or  no-go  torque‘s. 

Task  (1017) 

•  One  pilot  would  fly  the  other  would  check  systems  and  drift. 

•  Pilot  on  the  controls  remains  focused  outside  during  task. 

Task  (1018) 

•  One  pilot  would  fly  the  other  would  monitor  the  systems. 

Task  (1023) 

•  Pilot  on  the  controls  stays  outside.  Pilot  not  on  the  controls  notes  numbers  and  calculates 
numbers. 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

Task  (1025) 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

•  Pilot  on  the  controls  inboard  MFD  was  up  digital  map  for  scan  purposes.  Pilot  inboard 
MFD  was  switched  as  needed  by  the  mission. 

Task  (1026) 

•  As  long  as  data  and  flight  plans  were  loaded.  Pilot  entry  would  have  required  longer  run 
up. 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

Task  (1028) 

•  Pilot  on  the  controls  remained  outside  during  task,  pilot  not  on  the  controls  helped  as 
required. 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

Task  (1076) 
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•  One  pilot  would  fly  the  other  would  monitor  systems. 


Task  (1079) 

•  Pilot  handled  all  radios.  Pilot  on  the  controls  flew  the  aircraft. 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

Task  (1081) 

•  Pilot  on  the  controls  flew  the  aircraft.  Pilot  not  on  the  controls  stayed  ahead  of  aircraft  as 
much  as  possible,  performing  radio  communications  and  tuning  navigation  aids. 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

Task  (1082) 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

•  Pilot  on  the  controls  flew  aircraft.  Pilot  not  on  the  controls  helped  and  prompted  control 
inputs  while  scanning,  announced  descending  through  altitudes  for  arrival  at  FARP. 

Task  (1083) 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

•  Pilot  on  the  controls  initiated  IIMC  procedure,  flew  aircraft.  Pilot  not  on  the  controls 
backed  up  instrument  scan  for  first  2  minutes  then  squawked  EMER  and  made  radio 
calls,  tuned  VOR  and  gave  course  guidance  to  pilot  on  the  controls  to  proceed  direct  to 
the  VOR. 

•  Perfonning  inadvertent  IMC  procedures  is  difficult,  but  using  the  D-map  I  knew  where  I 
was. 

•  IIMC  is  a  high  workload  activity  regardless  of  aircraft. 

Task  (1135) 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

•  Once  the  controls  are  coupled  in  the  real  aircraft,  it  will  be  easier. 

Task  (1146) 

•  One  pilot  would  fly  the  other  would  monitor  systems. 

Task  (2009) 

•  A  Multi  ship  operation  is  a  high  workload  environment  and  there  is  no  difference 
between  the  two. 

Task  (2079) 

•  Under  NVG,  this  is  the  single  most  taxing  task  we  perform  in  current  aircraft.  In  this 
cockpit,  the  map  display  makes  this  an  easy  and  non-thinking  task. 

Task  (2081) 

•  Pilot  on  the  controls  scanned  outside.  Pilot  not  on  the  controls  stayed  outside,  task 
pennitting.  Low  bug  audio  on  radar  ALT  enhanced  situation  awareness  greatly. 
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Task  (2091) 

•  Sending  messages. 

•  Sending  tactical  reports  involves  a  lot  of  the  pilot’s  attention. 

•  It  still  takes  too  much  time  to  send  JVMF  messages. 


If  you  gave  a  workload  rating  of  6  or  higher  for  any  task  on  the  UH-60M  only,  explain  why  the 
workload  was  high  for  the  task. 

•  JVMF-  Too  many  things  to  do,  too  much  time  inside  the  cockpit.  The  pilots’  head  is 
down  to  send/receive/setup/change  nets. 

In  the  mission  you  just  flew,  list  any  flight  and/or  mission  tasks  on  the  UH-60M  that  you  had  to 
ask  your  crewmember  to  accomplish  because  your  workload  was  too  high. 

•  Call  -  ATC,  Mission  change  engage  a  change  to  route. 

•  Time  line  information 

•  Trouble  shooting  my  MFD/FMS,  but  only  due  to  my  lack  of  experience  with  a 
malfunctioning  MFDs. 

•  Make  some  radio  calls  because  I  was  busy  with  JVMF. 

•  Heading,  lighting  changes. 

•  Monitor  torque  during  IIMC. 
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Intentionally  left  blank 
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Appendix  H.  Pilot  SA  Comments 


Pilot  Situation  Awareness  Comments 

•  When  doing  JVMF  the  pilot  is  inside  and  only  one  pilot  is  outside.  The  pilot  inside  is 
looking  at  entering  /net  and  all  the  other  requirements  associated  with  JVMF. 

“Space”  on  entry  keypad  is  too  close,  spend  time  hunting  for  key’s  on  the  keyboard. 

•  Pilot  workload  is  great.  Change  of  mission  caught  my  attention.  Having  to  reset 
flight  plan. 

•  Aircraft  is  hot  with  the  glass  cockpit.  Needs  better  ventilation. 

•  PPC  that  is  shown  on  display. 

•  JVMF  messaging  is  very  tasking  compared  to  the  rest  of  the  aircraft. 

•  Trying  to  plot  threat  on  D-Map  as  waypoint. 

•  When  looking  at  the  route,  I  needed  to  look  at  the  infonnation  of  other  legs  on  the 
route. 

•  Getting  infonnation  on  threat  reported.  During  the  time  it  took  to  store  it  in  the  FMS 
and  locate  it  on  the  Digital  Map. 

•  IIMC  as  flight  S/A  of  where  other  aircraft  were. 

•  When  the  #2  FMS  locked  up,  I  didn’t  know  where  I  was. 
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Intentionally  left  blank 


62 


Appendix  I.  TSC  Comments 


TSC  MSI  Comments: 

•  The  mission  objectives  were  completed,  however  the  JVMF  time  synchronization 
made  it  more  difficult  to  complete  the  mission.  For  unknown  reason  there  is  a 
miscompare  between  the  XMIT  status  and  GPS  time  which  causes  the  JVMF  to 
fail. 

•  Went  into  dive  in  IIMC  and  crashed  aircraft.  FMS  #2  failure  contributed 
significantly  to  the  crash. 

•  The  crew  was  unable  to  maintain  aircraft  control  while  transitioning  to  IIMC. 

Pilot  workload  was  high  partly  due  to  excessive  power  and  pitch  changes 
resulting  in  zero  airspeed  and  nose  down  descent. 

•  Erroneous  information  contributed  to  crew  confusion  while  trying  to  maintain 
aircraft  control  while  IIMC.  FMS  failure  caused  the  pilot  not  on  the  controls  to  be 
unable  to  provide  the  pilot  on  the  controls  with  helpful  information  to  aid  in 
aircraft  recovery  from  an  unusual  attitude. 

TSC  MS2  Comments: 

•  Aborted  had  to  replace,  no  IFM. 

•  Unusual  attitude  trying  to  avoid  terrain.  Both  pilots  confused  about  what 
information  was  actually  correct. 

TSCWL1  Comments: 

•  Even  large  differences  in  level  of  proficiency  between  pilot  and  copilot  workload 
was  manageable. 

•  Could  not  keep  ahead  of  aircraft  and  mission  tasks. 

•  Workload  was  adequately  divided  between  the  two  aviators. 

•  During  the  mission  change  the  pilot  not  on  the  controls  was  required  to 
communicate  with  the  flight,  change  the  flight  plan  and  talk  voice  with  A2C2S. 
The  aviator  was  slow  to  prioritize  mission  task.  As  a  result  of  the  task  overload, 
the  initial  course  to  the  waypoint  was  vague.  Eventually  the  pilot  not  on  the 
controls  requested  assistance  from  the  pilot  on  the  controls. 

•  With  improved  proficiency  aviators  are  challenged  less  by  the  demands  of  the 
cockpit. 

•  The  crew  was  able  to  successfully  plot  the  intel  on  the  threat  in  flight.  The  time 
required  to  plot  the  threat  lead  the  crew  right  over  the  threat  before  they 
discovered  the  location  of  the  threat. 

•  One  pilot  was  continuously  behind  the  aircraft. 

•  IIMC  provided  task  overload  resulting  in  crash.  FMS  #2  failure  added  difficulty. 

•  Pilot  on  the  controls  was  unable  to  process  desired  pitch  attitude  setting  caused  by 
large  changes  in  power  setting  which  resulted  in  an  unusual  attitude  that  could  not 
be  recovered  from. 
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TSCWL2  Comments: 

•  Aircrew  worked  well  in  dividing  the  workload  and  communicating  actions  a 
majority  of  the  time.  The  crew  failed  to  verbalize  actions  in  off-tuning. 

•  NAVAIDS  on  a  few  occasions.  Overall  satisfactory  on  crew  coordination. 

•  Excellent  crew  coordination  during  IIMC  and  during  instrument  procedures  and 
approach.  Use  of  MFD  pages  excellent. 

•  Crew  worked  well  together  in  managing  the  flight  and  identifying  and  separating 
crew  responsibilities  of  the  pilot  on  the  controls  and  non-flying  pilot. 

•  Crew  coordination  was  overall  satisfactory.  At  one  point  during  the  mission 
change  prior  to  ACP10  both  crew  members  were  inside  briefly  sending  a  JVMF 
message  and  pilot  on  the  controls  changing  plight  plan  to  reflect  mission  change. 

•  Crew  coordination  was  below  previously  set  standard.  The  crew  failed  to  manage 
avionics  frequencies  IAW  the  execution  matrix. 

•  Crew  did  not  always  announce  actions  and  intentions. 

•  Excellent  crew  coordination.  The  left  and  right  digital  maps  had  dissimilar 
displays.  They  were  able  to  discuss  and  work  around  the  problem. 

•  Difficulty  troubleshooting  MFD/FMS  problem. 

•  Crew  coordination  failed  to  provide  SA  information  until  a  decision  was  made. 
Ex.  Departing  the  LZ  direct  to  the  FARP  was  planned,  but  not  executed.  There 
was  no  mention  of  the  change  until  the  PIC  announced  take  for  SP8. 

•  Good  crew  coordination  throughout  mission. 

•  Excellent  crew  coordination  for  diagnosing  the  EP  and  maintaining  aircraft 
control.  During  the  emergency  procedure  the  crew  climbed  approximately  2000 
ft  AGL.  This  is  contributed  to  the  long  approach  and  delay  in  landing  the  aircraft. 
Suspect  the  pilot  on  the  controls  was  distracted  by  the  EP. 

•  Greater  crew  coordination  may  have  resulted  in  a  successful  recovery  from  the 
unusual  attitude  with  the  non-flying  pilot  calling  out  pitch,  power,  trim,  and 
airspeed  to  aid  in  aircraft  recovery. 

TSC  SA  Comments: 

•  During  the  departure  procedure  the  crew  failed  to  maintain  course  alignment 
tracking  to  EMRUD.  The  crew  was  aware  of  the  error,  but  failed  to  make  the 
necessary  corrections.  Once  the  crew  was  vectored  for  correction  they  were  able 
to  regain  situational  awareness. 

•  Crew  briefly  lost  situational  awareness  while  perfonning  ILS  approach  into  Bike 
Lake.  The  loss  of  situational  awareness  was  contributed  to  misinterpretation  of 
approach  procedures. 

•  Tuned  wrong  COM  frequency  using  remote  collective  switch. 

•  Had  to  convert  threat  location  from  MGRS  to  Lat/Long. 

•  Departing  the  LZ  for  the  FARP,  the  crew  departed  to  the  northeast  instead  of 
southwest  to  the  FARP.  Aircrew  may  have  suffered  from  lack  of  exposure  to  the 
CTB.  The  crew  hadn’t  been  in  the  CTB  in  five  days. 

•  All  JVMF  MSG  &  were  viewed  as  “pending”,  during  later  of  mission  crew  asked 
for  acknowledge  for  position  reports. 

•  Last  JVMF  MSG  went  unnoticed.  No  visual  indicator.  Possible  instrumentation 
failure.  (Note:  Indicator  light  failure  previously  identified.) 


64 


•  At  one  point  Chalk  Three  reported  loss  of  visual  contact  with  Flight  One  and 
Two.  Was  unable  to  detennine  if  the  cause  was  due  to  separation  by  distance, 
being  obscured  by  terrain,  or  weather. 

•  Very  aware  of  terrain  in  poor  weather  with  digital  map. 
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66 


Appendix  J.  Pilot  PVI  Summary  and  Comments 


PV1: 


Pilot  PVI  Comments 


During  the  mission  did  you  experience  a  problem 
using  the  Multifunction  Displays  (MFD)? 


</) 
d> 
i /> 

Yes 

c 

o 

Q. 

</> 

No 

CD 

O' 

Not  Used 

91.7% 


0.0% 

0.0%  20.0%  40.0%  60.0%  80.0%  100.0% 


During  the  mission  did  you  experience  a  problem 
using  the  Primary  Flight  Display  (PFD)? 


During  the  mission  did  you  experience  a  problem 
using  the  Navigation  Display  (ND)? 
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During  the  mission  did  you  experience  a  problem 
using  the  Engine  Instrument  Caution  Advisory 
System  (EICAS)? 


During  the  mission  did  you  experience  a  problem 
using  the  Digital  Map  System  (DMS)? 


During  the  mission  did  you  experience  a  problem 
using  the  Joint  Variable  Message  Format  (JVMF)? 


0.0%  10.0%  20.0%  30.0%  40.0%  50.0%  60.0% 


During  the  mission  did  you  experience  a  problem 
using  the  Flight  Management  System  (FMS)? 


M) 

O 

</> 

Yes 

e 

o 

Q. 

</> 

No 

d> 

Not  Used 

j]  22.2% 
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During  the  mission  did  you  experience  a  problem 
using  the  JVMF  Entry? 


During  the  mission  did  you  experience  a  problem 
using  the  GPS/Flight  Plan? 
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During  the  mission  did  you  experience  a  problem 
using  the  Voice  Com  m  unications? 


During  the  mission  did  you  experience  a  problem 
using  the  Radio  Navigation? 
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During  the  mission  did  you  experience  a  problem 
using  the  Multifunction  Display  Slew  Controller 
(MFSC)? 
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During  the  mission  did  you  experience  a  problem 
using  the  Flight  Director  Display  Control  Panel 
(FDDCP)? 
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•  EI CAS  -  0-100%  scale  and  a  PPC  readout  would  be  great. 

•  JVMF  -  Just  too  difficult,  work  is  intense  and  the  head  is  inside. 

•  JVMF  -  Net  set  up  too  difficult. 

•  JVMF  -  Took  time  to  figure  out  why  a  message  “timed  out”  only  to  realize  that  a 
limitation  in  the  system  (Manually  entering  the  net)  led  to  user  error.  Should  look  for  a 
net  automatically  when  I  switch  to  another  data  frequency. 

•  JVMF  Entry  -  If  I  could  backspace  it  would  save  time.  Currently  if  I  need  to  backspace  a 
letter,  pushing  the  “CFR”  key  kills  my  errors  message. 

•  The  JVMF  needs  work.  I  needed  to  review  a  message  I  sent  and  there  was  no  easy  way 
to  do  it. 

•  JVMF  locked  up  once,  had  to  turn  it  off  and  then  reset. 

•  When  using  switches  T1-T6  my  arm  obstructs  the  view  of  the  MFD.  Recommend:  (Use 
F1-F6  or  R1-R6  for  JVMF  tasks). 

•  Incoming  message  reverse  video  does  not  get  my  attention  if  MFD  is  on  anything  other 
than  JVMF  page. 

•  You  can  easily  miss  an  incoming  message  because  there  is  no  way  to  notify  you. 

•  Digital  Map  -  Failure  of  RH  Pilot. 

•  MFD  -  At  one  point  my  MFD  began  feeding  false  information  concerning  my  course  and 
position.  Once  I  realized  I  could  isolate  my  MFD  to  the  other  MFD/FMS. 

•  FMS  -  When  I  reloaded  the  flight  plan,  my  FMS  would  not  update  my  position  correctly. 
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•  JVMF  -  At  one  point  I  could  not  get  JVMF  to  switch  to  a  different  net  until  I  reinitialized 
the  entire  JVMF. 

•  JVMF  was  time  consuming;  the  messages  that  were  sent  did  not  go  through.  Writing  free 
text  was  also  time  consuming. 

•  Several  times  during  the  mission  my  NAV  source  would  change  from  FMS  to  VOR.  It 
was  easy  to  change  back,  but  it  shouldn’t  have  changed. 

•  The  inbox  does  not  queue  the  crew  member  that  a  message  is  in  the  box.  An 
improvement  may  be  any  form  of  visual  or  audio  that  is  proactive  in  alerting  the  crew. 

•  JVMF  Entry  -  I  have  to  keep  hitting  “ENT”  on  the  FMS  when  I  mean  to  press  “SPACE” 
due  to  it’s  close  proximity  and  I  am  hurrying  to  relay  information. 

•  While  JVMF  XMIT  menu  is  up  and  a  Caution  alert  box  comes  up,  when  you  select 
“CNCL”  it  reverts  your  JVMF  page  to  SETUP. 

•  JVMF  messaging  could  be  simplified  by  being  able  to  do  more  while  pushing  fewer 
buttons  and  flipping  through  less  pages. 

•  EICAS  Page  -  If  system  pressure  or  temperature  goes  into  red,  symbology  needs  to  turn 
red. 

•  IDM  would  not  initialize  on  to  net.  Messages  stayed  in  pending  status. 

•  Too  much  time  is  needed  to  send  a  message  and  read  messages. 

•  The  #2  FMS  locked  up,  freezing  the  dig-map,  ARC  view,  and  ground  speed.  I  was 
unable  to  help  my  other  pilot  because  I  was  trying  to  fix  my  problem. 


PV2: 


On  average,  how  quickly  were  you  able  to  navigate 
through  menu  screens  on  the  Primary 
Flight/Navigaton  Displays  (PFD/ND)? 
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On  average,  how  quickly  were  you  able  to  navigate 
through  menu  screens  on  the  Digital  Map  System 
(DMS)? 
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On  average,  how  quickly  were  you  able  to  navigate 
through  menu  screens  on  the  Flight  Management 
System  (FMS)? 


m 

o 

in 

c 

o 

Q. 
in 
o 
0 £ 


Very  Quickly 
Somewhat  Quickly 
Borderline 
Somewhat  Slowly 
Very  Slowly 


30.6% 

) 

jj  bo .y  / 

05.6% 
1  0.0% 

I  0.0% 

r  r  r  r  r  ^ 

0.0%  20.0%  40.0%  60.0%  80.0%  100.0% 


On  average,  how  quickly  were  you  able  to  navigate 
through  menu  screens  on  the  Engine  Instrument 
Caution  Advisory  System  (EICAS)? 
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On  average,  how  quickly  were  you  able  to  navigate 
through  menu  screens  on  the  Joint  Variable 
Message  Format  (JVMF)? 
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•  JVMF  inability  to  join  a  new  TI  Net  automatically. 

•  Too  much  head  inside  time.  Must  set  up,  enter,  then  page  to  proper  places.  Message  and 
keyboarding  free  texts  takes  time. 

•  FMS  has  too  many  buttons. 

•  JVMF  has  too  many  screens  to  get  the  info  I  want. 

•  On  JVMF  many  of  the  most  used  buttons  are  on  the  top  and  you  have  to  move  your  arm 
each  time  you  use  them.  Look  at  putting  these  buttons  on  the  side. 

•  El  CAS  process  of  numbers  was  slow.  If  system  limits  were  set  at  0-100%  quick  view 
would  be  possible. 

•  JVMF  process  is  detailed  the  enter  key  when  selected  blocks  the  view  of  the  MFD  screen. 

•  EICAS  -  Didn’t  focus  fine  limit  #’s,  digital  readouts  good  just  limit  lines  hard  to  see 
Mopp/(No  Glasses) 

•  JVMF  -  Just  difficult  to  join  net  all  those  functions  should  be  automatic 

•  The  JVMF  pages  could  be  simplified  to  a  point  where  most  of  the  pertinent  information 
needed  for  it’s  use  and  operation  is  on  the  initial  page. 

•  JVMF  enter  button  is  at  the  top  left.  I  had  to  relocate  my  hand  to  view  screen  after 
making  a  selection. 

•  There  are  too  many  pages  on  the  JVMF  function. 
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PV3: 


How  often  did  you  forget  the  steps  required  for 
navigating  through  the  menu  screens  of  the  Primary 
Flight/Navigation  Displays  (PFD/ND)? 
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How  often  did  you  forget  the  steps  required  for 
navigating  through  the  menu  screens  of  the  Digital 
Map  System  (DMS)? 
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How  often  did  you  forget  the  steps  required  for 
navigating  through  the  menu  screens  of  the  Flight 
Management  System  (FMS)? 
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How  often  did  you  forget  the  steps  required  for 
navigating  through  the  menu  screens  of  the  Engine 
Instrument  Caution  Advisory  System  (EICAS)? 


How  often  did  you  forget  the  steps  required  for 
navigating  through  the  menu  screens  of  the  Joint 

Variable  Message  Format  (JVMF)? 
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•  Spending  time  looking  for  proper  sequences.  A  “Loop”  forms  where  I  can’t  get  to  what  I 
want,  I  have  to  stop  and  page  through  stuff. 

•  JVMF  inability  to  join  a  new  TI  Net  automatically. 

•  JVMF  -  Should  be  easier.  Set  frequency  and  system  finds  and  join’s  everything  like  cell 
phone. 

•  I  did  forget  how  to  navigate  through  the  GPS  waypoint  pages. 
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PV5: 


Did  you  have  any  difficulty  using  any  of  the  switches 
on  the  Collective  Grip? 


Did  you  have  any  difficulty  using  any  of  the  switches 
on  the  Cyclic  Grip? 
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•  These  sticks  are  very  intuitive  and  they  fit  your  hand  very  well. 

•  The  Radio  Remote  button  should  be  reversed  i.e.  push  up  to  scroll  up. 

•  Frequency  select  switch  on  the  collective  is  backwards.  To  change  frequencies  it  should 
be  push  up  to  scroll  up. 

•  MOPP  -  Gloves/  Pilot  gloves  size  12  pilots-  just  not  big  enough  /  Large  MOPP  -  just  not 
big  enough 

•  The  preset  change  switch  on  the  collective  seems  to  be  backwards,  (ie:  “UP”  moves  the 
preset  down  and  vice  versa) 

•  The  preset  comm.  Frequency  slew  switch  on  the  collective  increases  the  preset  number 
when  it  is  pulled  back.  This  is  backwards  in  my  crew  procedures.  [Forward  =  UP  and 
Back  =  Down]  to  me. 

•  While  in  MOPP  4,  my  right  hand  became  fatigued  and  sensitivity  was  lost  during  the 
mission.  I  had  problems  placing  my  hand  comfortably  on  the  cyclic  especially  with  the 
cyclic  mounted  stabilator  slew-up  switch. 

•  The  freq  select  switch  is  wrong. 

•  Collective  Freq  select  switch  as  before  on  all  missions. 
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During  MOPP  4,  three  layers  of  gloves  made  the  cyclic  cramped. 


PV6: 


Was  there  any  symbology  depicted  on  the  Primary 
Flight  Displays  (PFD)  that  was  difficult  to  understand? 


Was  there  any  symbology  depicted  on  the  Navigation 
Display  (ND)  that  was  difficult  to  understand? 
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Was  there  any  sym  bology  on  the  Engine  Instrument 
Caution  Advisory  System  (EICAS)  that  was  difficult  to 
understand? 
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Was  there  any  symbology  on  the  Digital  Map  System 
(DMS)  that  was  difficult  to  understand? 


Was  there  any  symbology  on  the  Aircraft  Survivability 
Equip  (ASE)  that  was  difficult  to  understand? 
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•  I  have  to  closely  look  at  the  scale  marking  0-100%.  All  scales  give  growth. 

•  El  CAS:  system  limits  should  be  based  on  0-100%  (Green/Yellow/Red)  not  specific 
numbers. 

•  El  CAS  -  Limit  lines  hard  to  see,  0-100%  for  all 

•  ND  /  DMS  -  When  my  MFD/FMS  malfunctioned  it  took  me  some  time  to  remember 
how  to  isolate  to  the  other  system  since  I  had  not  experience  this  failure  before. 

•  While  using  the  gas  mask.  I  had  to  make  a  more  noticeable  effort  to  tilt  my  head  down  to 
scan  the  current  G/S  readout.  Placing  the  G/S  a  little  higher  on  the  MFD/PFD  would  take 
care  of  this. 

•  0-100%.  Chapter  5  limits  are  not  really  required  to  be  memorized. 

•  Yellow  scale  lines  tended  to  mix  with  yellow  scale.  (Proposed  fix:  Make  a  box  around 
the  yellow  limit  and  let  scale  move  through.) 

•  Aircraft  icon  on  ND  screen  needs  to  be  a  color  other  than  white  so  that  it  does  not  blend 
with  colors  on  the  map. 


PV7: 


Did  you  experience  any  problems  with  symbology 
clutter  on  the  PFD-Full  display? 


Did  you  experience  any  problems  with  symbology 
clutter  on  the  PFD-Arc  display? 
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Did  you  experience  any  problems  with  symbology 
clutter  on  the  PFD-Hover  display? 


Did  you  experience  any  problems  with  symbology 
clutter  on  the  ND-Full  display? 


Did  you  experience  any  problems  with  symbology 
clutter  on  the  ND-Plan  display? 
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•  ND  /  DMS  -  When  my  MFD/FMS  malfunctioned  it  took  me  some  time  to  remember 
how  to  isolate  to  the  other  system  since  I  had  not  experience  this  failure  before. 

•  I  don’t  think  the  hover  page  help  much. 

•  On  the  D-map,  labels  would  block  each  other  from  viewing  and  the  D-map  scale  goes 
from  10  to  20.  Would  be  nice  to  have  a  15  in  there. 

•  When  scaled  out  to  20  on  the  radius  ring  labels  overwrite  each  other.  Make  it  so  they 
don’t  overwrite  each  other. 

•  When  the  #2  FMS  locked  up,  I  lost  my  digmap,  ARC  view,  and  ground  speed  on  my 
MFD. 

PV8: 


Did  you  experience  any  sort  of  hand  discomfort  using 
the  MFSC  grip? 
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Did  you  experience  any  sort  of  hand  discomfort  using 
the  Collective  grip? 


•  The  weapons  release  cover  gets  in  the  way  of  cyclic  stick  trim  switch  to  the  right. 

•  Yes,  Due  to  MOPP  4  Level. 

•  My  right  hand  barely  fit  around  the  cyclic  with  3  layers  of  gloves  and  was  cramped  by 
the  cyclic  mounted  stabilator  slew-up  switch. 

•  MOPP  4  with  3  layers  of  gloves  -  the  cyclic  is  small  and  causes  difficulty. 
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For  the  JVMF  reports,  how  would  you  rate  the 
ease/difficulty  of  sending  the  Position  Report? 


L 


For  the  JVMF  reports,  how  would  you  rate  the 
ease/difficulty  of  sending  the  Free  Text  Message 
report? 


Too  much  work  inside. 

Too  easy  to  select  the  wrong  recipient. 

Backspace  versus  “CLR”  ability  when  typing. 

It’s  a  high  workload  activity  to  punch  in  buttons  and  send  messages.  There  are  too  many 
steps  involved. 

Free  text  messages  scroll  down  to  free  text  then  confirm  then  enter. 

I  should  be  able  to  select  free  text  then  type. 

It  takes  time  to  fat  finger  text. 

Location  of  enter  key,  the  ann  blocks  the  screen.  Time  required  to  compute  message,  one 
crewmember  is  inside  for  long  times. 

MOPP  gloves  were  tight  and  wet  inside  liners 

Position  reports  could  be  cut  down  to  a  1  or  2  step  process  free  text  messaging  takes  too 
much  time. 

Having  3  layers  of  gloves  on  made  me  loose  a  little  sensitivity  of  the  lingers. 

I  spent  a  little  more  time  pushing  buttons. 

JVMF  should  set  up  radios/nets. 
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•  Keyboard  space  -  Space  and  Enter  keys  are  too  close. 

•  Free  Text  -  When  I  am  hurrying,  I  continuously  press  “ENT”  when  I  mean  to  press 
“SPACE”  due  to  the  close  proximity  of  the  buttons. 

•  Free  Text  -  Because  of  3  layers  of  gloves,  loss  of  sensitivity  of  fingers  extra  time  was 
used  to  ensure  accuracy  of  which  buttons  were  pushed. 

•  Too  much  time  spent  inside  the  cockpit. 

•  Free  text  keeps  one  pilot  “heads  down”  inside  for  several  minutes  at  a  time. 

•  Net  entry  should  be  easier  (like  a  cell  phone). 

•  Should  be  easier,  less  button  presses  -  like  bezzle  button  marked  “position  report”  press 
it  twice  and  send. 

•  Freetext  -  Slow  and  head  down  inside  to  do  this. 

•  Too  much  time  spent  inside  sending  and  typing  reports. 


PV10: 


How  would  you  rate  your  ability  to  detect  the  following 
occurrences  based  on  the  characteristics  of  the  flight 
displays?  -  JVMF  Message  (MFD) 
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How  would  you  rate  your  ability  to  detect  the  following 
occurrences  based  on  the  characteristics  of  the  flight 
displays?  -  Caution/Advisory  (MFD) 
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How  would  you  rate  your  ability  to  detect  the  following 
occurrences  based  on  the  characteristics  of  the  flight 
displays?  -  Warning  (Master  Warning  Panel) 


U) 

<u 

in 

c 

o 

Q. 

in 

<u 

a. 


Very  Easy 
Somewhat  Easy 
Borderline 
Somewhat  Difficult 
Very  Difficult 


58.3% 


r  r-  f  t"  f  f 

0.0%  20.0%  40.0%  60.0%  80.0%  100.0% 


How  would  you  rate  your  ability  to  detect  the  following 
occurrences  based  on  the  characteristics  of  the  flight 
displays?  -  Entry  Into  Operational  Limits  (per  Chp  5)  on 
the  Power  Rod 


•  Message  alert  is  not  easily  recognizable. 

•  JVMF  inverse  video  is  inadequate  -  Suggest  a  limited  border  on  the  MFD  that  can  be 
acknowledged. 

•  The  JVMF  indication  was  not  big  enough 

•  Reverse  video  on  any  page  other  than  JVMF  is  not  enough  to  catch  my  attention. 

•  You  cannot  tell  when  you  have  an  incoming  message. 

•  I  have  never  seen  the  JVMF  reverse  video. 

•  JVMF  -  Just  hard  to  see  incoming  messages. 

•  JVMF  -  Inverse  video  is  not  easily  recognizable. 

•  There  is  not  a  good  indication  of  when  a  JVMF  message  has  come  to  your  inbox. 

•  Difficult  to  ID  JVMF. 

•  JVMF  -  Inverse  video  does  not  stand  out  enough  to  attract  attention. 

•  Reverse  video  on  incoming  messages. 

•  The  JVMF  indication  of  an  incoming  message  is  not  adequate. 

•  There  is  no  good  indication  of  an  incoming  message. 

•  No  visible  inbox  when  you  have  mail  alert. 

•  JVMF  -  Light  is  hard  to  see,  need  better  attention  getter. 
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•  Operating  Limits  -  They  are  in  the  system  and  pilot  no  longer  needs  to  have  memorized. 

•  JVMF  -  Receiving  a  message  is  difficult  to  detect.  Inverse  video  on  JVMF  is  not  enough 
to  get  your  attention. 

•  Operational  limits  -  after  receiving  a  “Check  EICAS”  message  I  could  not  easily 
determine  what  the  problem  was  until  I  noticed  a  change  in  digital  number  readout. 
Additionally,  if  the  precautionary  line  changed  to  inverse  video  when  the  tape  changed  to 
yellow,  I  could  have  more  easily  determined  how  far  into  that  range  I  was. 

•  The  JVMF  indication  should  be  easier  to  detect.  An  outline  on  the  inboard  MFD  that 
lights  up  when  a  message  is  received  would  be  easier  to  detect. 

•  There  is  not  a  clean  indication  of  an  incoming  message. 

•  JVMF  inbox  as  ALL  missions  -  the  queue  for  inbox  message  is  not  visible. 

•  JVMF  -  Inverse  video  is  difficult  to  detect. 

•  Hard  to  identify  message  in  inbox. 

•  Inverse  video  on  incoming  message  is  hard  to  detect  unless  up  on  the  Inbox  screen. 

•  The  indication  of  a  message  should  be  an  outline  on  the  inboard  MFD  that  gives  an 
option  of  coming  back  to  check  the  message  or  to  check  it. 

•  There  is  no  good  indication  of  an  incoming  message. 

•  No  visual  queue  for  an  incoming  message. 
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Out  The  Window 


Left  Seat 

82.23%  (VFR/FP/AM) 
17.44%  (VFR/NFP/AM) 


Right  Seat 

88.50%  (VFR/FP/AM) 
41.71%  (VFR/NFP/AM) 


89.32%  (VFR/FP/MSM) 
19.13%  (VFR/NFP/MSM) 


90.84%  (VFR/FP/MSM) 
42.92%  (VFR/NFP/MSM) 


50.00%  (VFR/FP/SSM) 
16.43%  (VFR/NFP/SSM) 


80.44  (VFR/FP/SSM) 
39.44%  (VFR/NFP/SSM) 


oo 

-a 


LMFD 

8.49%  (VFR/FP/AM) 
25.60%  (VFR/NFP/AM) 


RMFD 

6.10%  (VFR/FP/AM) 
32.49%  (VFR/NFP/AM) 


LMFD 

1.86%  (VFR/FP/AM) 
15.47%  (VFR/NFP/AM) 


RMFD 

5.79%  (VFR/FP/AM) 
18.76%  (VFR/NFP/AM) 


5.14%  (VFR/FP/MSM) 
24.16%  (VFR/NFP/MSM) 


4.89%  (VFR/FP/MSM) 
29.88%  (VFR/NFP/MSM) 


2.02%  (VFR/FP/MSM) 
13.47%  (VFR/NFP/MSM) 


3.51%  (VFR/FP/MSM) 
12.64%  (VFR/NFP/MSM) 


23.73%  (VFR/FP/SSM) 
26.46%  (VFR/NFP/SSM) 


11.64%  (VFR/FP/SSM) 
34.05%  (VFR/NFP/SSM) 


1.31%  (VFR/FP/SSM) 
19.22%  (VFR/NFP/SSM) 


13.63%  (VFR/FP/SSM) 
30.22%  (VFR/NFP/SSM) 


Other 


3.17%  (VFR/FP/AM) 
24.47%  (VFR/NFP/AM) 


0.65%  (VFR/FP/MSM) 
26.82%  (VFR/NFP/MSM) 


‘Other’  =  primarily  the  area  below  the 
MFDs  (e.g.,  FMS,  kneeboard). 


Other 


3.85%  (VFR/FP/AM) 
24.06%  (VFR/NFP/AM) 


3.63%  (VFR/FP/MSM) 
30.97%  (VFR/NFP/MSM) 


14.64%  (VFR/FP/SSM) 
23.06%  (VFR/NFP/SSM) 


4.63%  (VFR/FP/SSM) 
11.11%  (VFR/NFP/SSM) 


FP  =  Flying  Pilot,  NFP  =  Non-Flying  Pilot,  VFR  =  Visual  Flight  Rules,  AM  =  All  Missions,  MSM  =  Multi-ship  Missions,  SSM  =  Single-Ship 
Mission,  VFR  Missions  (single  &  multi-ship)  =  7  missions  (13.40  hours  of  data),  VFR  Missions  (multi-ship  only)  =  5  missions  (8.63  hours  of 
data),  VFR  Missions  (single-ship  only)  =  2  missions  (4.76  hours  of  data),  IFR  data  (Vignette  1)  not  valid. 


Appendix  K.  Eye  Tracker  Results  Summary 


Intentionally  left  blank 


88 


Glossary  of  Acronyms 


AAR 

after-action  review 

ACP 

air  control  point 

AOI 

area  of  interest 

ARL 

Army  Research  Laboratory 

ASL 

Applied  Science  Laboratories 

ATM 

aircrew  training  manual 

BHIVE 

Battlefield  Highly  Immersive  Virtual  Environment 

BWRS 

Bedford  Workload  Rating  Scale 

CAAS 

Common  Avionics  Architecture  System 

CPC 

Comanche  portable  cockpit 

CW 

Chief  Warrant 

DCD 

Directorate  of  Combat  Developments 

DIGMAP 

digital  mapping  system 

EDS 

engineering  development  simulator 

EICAS 

engine  instrument  caution  advisory  system 

EUD 

early  user  demonstration 

FMS 

flight  management  system 

HFE 

human  factors  engineering 

HRED 

Human  Research  and  Engineering  Directorate 

IFR 

instrument  flight  rules 

IIMC 

inadvertent  instrument  meteorological  conditions 

IMC 

instrumented  meteorological  conditions 

JVMF 

joint  variable  message  format 

LEUE 

limited  early  user  evaluation 

LUT 

limited  user  test 

MANPRINT 

manpower  and  personnel  integration 

MEDEVAC 

medical  evacuation 

MFD 

multi-function  display 

MP 

mission  profile 

OMS 

operational  mode  summary 

OTW 

out  the  window 

PFD 

primary  flight  display 

PM 

product  manager 

PMO 

Product  Manager’s  Office 

PVI 

pilot-vehicle  interface 

SA 

situational  awareness 

SART 

Situational  Awareness  Rating  Technique 

SED 

Software  Engineering  Directorate 
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SIL 

System  Integration  Laboratory 

SSQ 

Simulator  Sickness  Questionnaire 

TRADOC 

Training  and  Doctrine  Command 

TSC 

tactical  steering  committee 

TSM 

TRADOC  System  Manager 

UH 

utility  helicopter 

VFR 

visual  flight  rules 

VMC 

visual  meteorological  conditions 

V&V 

verification  and  validation 

WSRT 

Wilcoxon  Signed  Ranks  Test 

XP 

experimental  test  pilot 
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COPIES  ORGANIZATION 


DEFENSE  TECHNICAL 
INFORMATION  CTR 
DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

US  ARMY  RSRCH  DEV  &  ENGRG  CMD 

SYSTEMS  OF  SYSTEMS 

INTEGRATION 

AMSRD  SS  T 

6000  6TH  ST  STE  100 

FORT  BELVOIR  VA  22060-5608 

INST  FOR  ADVNCD  TCHNLGY 
THE  UNIV  OF  TEXAS  AT  AUSTIN 
3925  W  BRAKER  LN  STE  400 
AUSTIN  TX  78759-5316 

DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  IMS 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRD  ARL  Cl  OK  TL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRD  ARL  CS  OK  T 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

DIRECTOR 

UNIT  OF  ACTION  MANEUVER  BATTLE  LAB 
ATTN  ATZKUA 
BLDG  1101 

FORT  KNOX  KY  40121 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  M  DR  M  STRUB 
6359  WALKER  LANE  SUITE  100 
ALEXANDRIA  VA  22310 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MA  J  MARTIN 
MYER  CENTER  RM  2D3 1 1 
FT  MONMOUTH  NJ  07703-5630 


1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MC  A  DAVISON 
320  MANSCEN  LOOP  STE  166 
FT  LEONARD  WOOD  MO  65473-8929 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MD  T  COOK 
BLDG  5400  RM  C242 
REDSTONE  ARSENAL  AL  35898-7290 

1  COMMANDANT  USAADASCH 
ATTN  ATS  A  CD 

ATTN  AMSRD  ARL  HR  ME  MS  A  MARES 
5800  CARTER  RD 
FT  BLISS  TX  79916-3802 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MI  J  MINNINGER 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MM  DR  V  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 
FT  SAM  HOUSTON  TX  78234-5094 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MG  R  SPINE 
BUILDING  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARL  HRED  ARMC  FLD  ELMT 

ATTN  AMSRD  ARL  HR  MH  C  BURNS 
BLDG  1467B  ROOM  336 
THIRD  AVENUE 
FT  KNOX  KY  40121 

10  ARMY  RSCH  LABORATORY  -  HRED 
AVNC  FIELD  ELEMENT 
ATTN  AMSRD  ARL  HR  MJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MK  MR  J  REINHART 
10125  KINGMAN  RD 
FT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MV  HQ  USAOTC 
S  MIDDLEBROOKS 
91012  STATION  AVE  ROOM  111 
FT  HOOD  TX  76544-5073 
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ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MY  M  BARNES 
2520  HEAL Y  AVE  STE  1 1 72  BLDG  5 1 005 
FT  HUACHUCA  AZ  85613-7069 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MP  D  UNGV ARSKY 
BATTLE  CMD  BATTLE  LAB 
415  SHERMAN  AVE  UNIT  3 
FT  LEAVENWORTH  KS  66027-2326 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MJK  J  HANSBERGER 
JFCOM  JOINT  EXPERIMENTATION  J9 
JOINT  FUTURES  LAB 
115  LAKEVIEW  PARKWAY  SUITE  B 
SUFFOLK  VA  23435 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MQ  M  R  FLETCHER 
US  ARMY  SBCCOM  NATICK  SOLDIER  CTR 
AMSRD  NSC  SSE  BLDG  3  RM  341 
NATICK  MA  01760-5020 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MY  DR  J  CHEN 
12423  RESEARCH  PARKWAY 
ORLANDO  FL  32826 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MS  MR  C  MANASCO 
SIGNAL  TOWERS  RM  303A 
FORT  GORDON  GA  30905-5233 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MU  M  SINGAPORE 
6501  E  11  MILE  RD  MAIL  STOP  284 
BLDG  200A  2ND  FL  RM  2104 
WARREN  MI  48397-5000 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MF  MR  C  HERNANDEZ 
BLDG  3040  RM  220 
FORT  SILL  OK  73503-5600 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MW  E  REDDEN 
BLDG  4  ROOM  332 
FT  BENNING  GA  31905-5400 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MN  R  SPENCER 
DCSFDI  HF 

HQ  USASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 


1  ARMY  G1 

ATTN  DAPE  MR  B  KNAPP 

300  ARMY  PENTAGON  ROOM  2C489 

WASHINGTON  DC  20310-0300 

1  US  ARMY  SAFETY  CTR 
ATTN  CSSC  SE 
FORT  RUCKER  AL  36362 

1  MICRO  ANALYSIS  &  DESIGN  INC 
ATTN  BETHPLOTT 
4949  PEARL  E  CR  #300 
BOULDER  CO  80301 

ABERDEEN  PROVING  GROUND 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  Cl  OK  TECH  LIB 
BLDG  4600 

1  US  ATEC 
RYAN  BLDG 
APG-AA 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 

ATTN  AMSRD  ARL  Cl  OK  TP  S  FOPPIANO 

BLDG  459 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  HR  M  F  PARAGALLO 
BLDG  459 
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